Nov 01 06:09:09.495144 ip-10-0-140-68 systemd[1]: Starting Kubernetes Kubelet... Nov 01 06:09:09.857922 ip-10-0-140-68 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 06:09:09.857922 ip-10-0-140-68 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 06:09:09.857922 ip-10-0-140-68 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 06:09:09.857922 ip-10-0-140-68 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 06:09:09.862278 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862164 2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:09.862278 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862277 2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:09.862278 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862282 2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:09.862385 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862286 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:09.862385 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862289 2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:09.862385 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862292 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:09.862385 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862295 2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:09.862385 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862298 2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:09.862385 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862308 2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:09.862385 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862312 2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:09.862385 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862315 2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:09.862385 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862317 2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:09.862385 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862321 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:09.862385 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862323 2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:09.862385 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862326 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:09.862385 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862329 2567 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:09.862385 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862332 2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:09.862385 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862335 2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:09.862385 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862337 2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:09.862385 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862340 2567 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:09.862385 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862342 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:09.862385 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862345 2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:09.862385 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862350 2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:09.862845 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862353 2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:09.862845 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862356 2567 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:09.862845 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862359 2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:09.862845 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862362 2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:09.862845 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862365 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:09.862845 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862368 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:09.862845 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862370 2567 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:09.862845 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862373 2567 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:09.862845 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862376 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:09.862845 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862378 2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:09.862845 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862381 2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:09.862845 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862384 2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:09.862845 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862387 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:09.862845 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862390 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:09.862845 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862392 2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:09.862845 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862395 2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:09.862845 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862398 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:09.862845 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862401 2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:09.862845 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862405 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:09.862845 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862407 2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:09.863317 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862410 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:09.863317 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862413 2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:09.863317 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862415 2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:09.863317 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862418 2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:09.863317 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862421 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:09.863317 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862423 2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:09.863317 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862426 2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:09.863317 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862428 2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:09.863317 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862431 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:09.863317 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862433 2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:09.863317 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862436 2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:09.863317 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862440 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:09.863317 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862443 2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:09.863317 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862446 2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:09.863317 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862448 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:09.863317 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862451 2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:09.863317 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862454 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:09.863317 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862456 2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:09.863317 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862459 2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:09.863775 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862462 2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:09.863775 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862465 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:09.863775 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862468 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:09.863775 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862471 2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:09.863775 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862474 2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:09.863775 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862477 2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:09.863775 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862479 2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:09.863775 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862482 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:09.863775 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862485 2567 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:09.863775 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862488 2567 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:09.863775 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862490 2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:09.863775 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862493 2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:09.863775 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862496 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:09.863775 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862499 2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:09.863775 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862501 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:09.863775 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862504 2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:09.863775 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862506 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:09.863775 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862509 2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:09.863775 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862511 2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:09.863775 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862514 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:09.864267 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862517 2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:09.864267 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862519 2567 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:09.864267 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862522 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:09.864267 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862524 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:09.864267 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862527 2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:09.864267 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862529 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:09.864267 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862532 2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:09.864267 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862535 2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:09.864267 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862538 2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:09.864267 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862541 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:09.864267 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862544 2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:09.864267 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862546 2567 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:09.864267 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862550 2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:09.864267 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862553 2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:09.864267 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862555 2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:09.864267 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862558 2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:09.864267 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862561 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:09.864267 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862564 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:09.864267 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862566 2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:09.864267 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862569 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:09.864767 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862572 2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:09.864767 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862574 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:09.864767 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862577 2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:09.864767 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.862579 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:09.864767 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863450 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:09.864767 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863459 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:09.864767 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863463 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:09.864767 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863466 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:09.864767 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863469 2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:09.864767 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863472 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:09.864767 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863474 2567 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:09.864767 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863477 2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:09.864767 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863480 2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:09.864767 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863483 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:09.864767 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863485 2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:09.864767 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863488 2567 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:09.864767 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863491 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:09.864767 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863493 2567 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:09.864767 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863496 2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:09.865247 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863499 2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:09.865247 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863502 2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:09.865247 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863505 2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:09.865247 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863507 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:09.865247 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863511 2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:09.865247 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863513 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:09.865247 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863516 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:09.865247 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863518 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:09.865247 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863521 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:09.865247 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863525 2567 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:09.865247 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863528 2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:09.865247 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863531 2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:09.865247 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863534 2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:09.865247 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863537 2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:09.865247 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863539 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:09.865247 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863542 2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:09.865247 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863546 2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:09.865247 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863550 2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:09.865247 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863554 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:09.865718 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863557 2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:09.865718 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863560 2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:09.865718 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863563 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:09.865718 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863566 2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:09.865718 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863568 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:09.865718 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863571 2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:09.865718 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863573 2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:09.865718 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863576 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:09.865718 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863578 2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:09.865718 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863581 2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:09.865718 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863584 2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:09.865718 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863586 2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:09.865718 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863589 2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:09.865718 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863592 2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:09.865718 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863595 2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:09.865718 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863598 2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:09.865718 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863600 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:09.865718 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863603 2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:09.865718 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863606 2567 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:09.866206 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863610 2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:09.866206 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863614 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:09.866206 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863617 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:09.866206 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863620 2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:09.866206 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863623 2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:09.866206 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863625 2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:09.866206 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863628 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:09.866206 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863631 2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:09.866206 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863633 2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:09.866206 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863636 2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:09.866206 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863639 2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:09.866206 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863641 2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:09.866206 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863644 2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:09.866206 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863647 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:09.866206 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863651 2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:09.866206 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863654 2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:09.866206 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863656 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:09.866206 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863659 2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:09.866206 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863661 2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:09.866778 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863664 2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:09.866778 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863667 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:09.866778 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863669 2567 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:09.866778 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863672 2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:09.866778 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863674 2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:09.866778 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863677 2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:09.866778 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863680 2567 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:09.866778 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863682 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:09.866778 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863685 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:09.866778 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863688 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:09.866778 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863690 2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:09.866778 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863693 2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:09.866778 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863696 2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:09.866778 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863698 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:09.866778 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863701 2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:09.866778 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863703 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:09.866778 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863707 2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:09.866778 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863709 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:09.866778 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863712 2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:09.866778 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863715 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:09.867318 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863718 2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:09.867318 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863720 2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:09.867318 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863723 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:09.867318 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863726 2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:09.867318 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863728 2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:09.867318 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863731 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:09.867318 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863734 2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:09.867318 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863738 2567 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:09.867318 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863741 2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:09.867318 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863743 2567 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:09.867318 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863746 2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:09.867318 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863748 2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:09.867318 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863751 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:09.867318 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.863754 2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:09.867318 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864316 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:09.867318 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864324 2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:09.867318 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864327 2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:09.867318 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864332 2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:09.867318 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864336 2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:09.867766 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864339 2567 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:09.867766 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864342 2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:09.867766 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864345 2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:09.867766 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864348 2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:09.867766 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864351 2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:09.867766 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864354 2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:09.867766 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864357 2567 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:09.867766 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864360 2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:09.867766 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864363 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:09.867766 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864366 2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:09.867766 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864369 2567 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:09.867766 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864371 2567 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:09.867766 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864374 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:09.867766 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864377 2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:09.867766 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864380 2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:09.867766 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864383 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:09.867766 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864386 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:09.867766 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864388 2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:09.867766 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864391 2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:09.867766 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864393 2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:09.868254 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864396 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:09.868254 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864400 2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:09.868254 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864403 2567 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:09.868254 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864405 2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:09.868254 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864408 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:09.868254 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864411 2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:09.868254 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864413 2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:09.868254 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864416 2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:09.868254 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864418 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:09.868254 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864421 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:09.868254 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864424 2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:09.868254 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864427 2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:09.868254 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864430 2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:09.868254 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864432 2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:09.868254 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864435 2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:09.868254 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864437 2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:09.868254 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864440 2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:09.868254 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864442 2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:09.868254 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864445 2567 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:09.868254 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864448 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:09.868721 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864451 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:09.868721 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864454 2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:09.868721 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864457 2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:09.868721 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864460 2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:09.868721 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864462 2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:09.868721 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864465 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:09.868721 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864467 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:09.868721 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864470 2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:09.868721 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864472 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:09.868721 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864475 2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:09.868721 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864477 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:09.868721 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864480 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:09.868721 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864482 2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:09.868721 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864485 2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:09.868721 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864488 2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:09.868721 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864491 2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:09.868721 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864494 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:09.868721 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864496 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:09.868721 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864499 2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:09.869202 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864501 2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:09.869202 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864504 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:09.869202 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864506 2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:09.869202 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864509 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:09.869202 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864513 2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:09.869202 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864517 2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:09.869202 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864520 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:09.869202 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864523 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:09.869202 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864525 2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:09.869202 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864528 2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:09.869202 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864530 2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:09.869202 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864533 2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:09.869202 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864538 2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:09.869202 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864541 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:09.869202 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864544 2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:09.869202 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864546 2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:09.869202 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864549 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:09.869202 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864552 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:09.869202 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864555 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:09.869649 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864557 2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:09.869649 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864560 2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:09.869649 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864563 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:09.869649 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864566 2567 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:09.869649 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864568 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:09.869649 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864571 2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:09.869649 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864574 2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:09.869649 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864576 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:09.869649 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864579 2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:09.869649 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864582 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:09.869649 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864585 2567 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:09.869649 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864588 2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:09.869649 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864590 2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:09.869649 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864593 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:09.869649 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864596 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:09.869649 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864598 2567 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:09.869649 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864601 2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:09.869649 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864606 2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:09.869649 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864608 2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:09.869649 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864611 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864614 2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864616 2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.864619 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865523 2567 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865534 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865544 2567 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865549 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865554 2567 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865558 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865563 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865568 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865571 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865574 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865579 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865582 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865586 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865589 2567 flags.go:64] FLAG: --cgroup-root="" Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865592 2567 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865595 2567 flags.go:64] FLAG: --client-ca-file="" Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865598 2567 flags.go:64] FLAG: --cloud-provider="external" Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865602 2567 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865607 2567 flags.go:64] FLAG: --cluster-domain="" Nov 01 06:09:09.870141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865616 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865619 2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865622 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865626 2567 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865631 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865634 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865637 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865641 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865645 2567 flags.go:64] FLAG: --contention-profiling="false" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865648 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865651 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865654 2567 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865658 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865662 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865666 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865670 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865673 2567 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865676 2567 flags.go:64] FLAG: --enable-server="true" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865680 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865684 2567 flags.go:64] FLAG: --event-burst="100" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865688 2567 flags.go:64] FLAG: --event-qps="50" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865691 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865694 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865697 2567 flags.go:64] FLAG: --eviction-hard="" Nov 01 06:09:09.870682 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865701 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865705 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865708 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865712 2567 flags.go:64] FLAG: --eviction-soft="" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865715 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865718 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865722 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865725 2567 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865729 2567 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865738 2567 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865741 2567 flags.go:64] FLAG: --feature-gates="" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865747 2567 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865750 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865754 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865757 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865761 2567 flags.go:64] FLAG: --healthz-port="10248" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865764 2567 flags.go:64] FLAG: --help="false" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865767 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865771 2567 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865774 2567 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865777 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865780 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865784 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 06:09:09.871338 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865789 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865792 2567 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865795 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865798 2567 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865801 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865804 2567 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865807 2567 flags.go:64] FLAG: --kube-reserved="" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865810 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865813 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865817 2567 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865820 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865823 2567 flags.go:64] FLAG: --lock-file="" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865826 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865844 2567 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865848 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865854 2567 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865857 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865861 2567 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865864 2567 flags.go:64] FLAG: --logging-format="text" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865868 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865872 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865875 2567 flags.go:64] FLAG: --manifest-url="" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865878 2567 flags.go:64] FLAG: --manifest-url-header="" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865883 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865886 2567 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 06:09:09.871926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865890 2567 flags.go:64] FLAG: --max-pods="110" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865893 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865897 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865900 2567 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865903 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865906 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865909 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865914 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865922 2567 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865925 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865929 2567 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865932 2567 flags.go:64] FLAG: --pod-cidr="" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865935 2567 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865938 2567 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865941 2567 flags.go:64] FLAG: --pods-per-core="0" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865944 2567 flags.go:64] FLAG: --port="10250" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865947 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865952 2567 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-080efc09aabf5cbfe" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865955 2567 flags.go:64] FLAG: --qos-reserved="" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865958 2567 flags.go:64] FLAG: --read-only-port="10255" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865961 2567 flags.go:64] FLAG: --register-node="true" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865964 2567 flags.go:64] FLAG: --register-with-taints="" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865972 2567 flags.go:64] FLAG: --registry-burst="10" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865975 2567 flags.go:64] FLAG: --registry-qps="5" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865978 2567 flags.go:64] FLAG: --reserved-cpus="" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865981 2567 flags.go:64] FLAG: --reserved-memory="" Nov 01 06:09:09.872544 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865986 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865990 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865993 2567 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865996 2567 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.865999 2567 flags.go:64] FLAG: --runonce="false" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866002 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866006 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866009 2567 flags.go:64] FLAG: --seccomp-default="false" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866012 2567 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866016 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866019 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866022 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866026 2567 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866030 2567 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866033 2567 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866036 2567 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866039 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866042 2567 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866046 2567 flags.go:64] FLAG: --system-cgroups="" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866048 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866054 2567 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866057 2567 flags.go:64] FLAG: --tls-cert-file="" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866062 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866067 2567 flags.go:64] FLAG: --tls-min-version="" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866070 2567 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 06:09:09.873243 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866073 2567 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 06:09:09.874216 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866076 2567 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 06:09:09.874216 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866080 2567 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 06:09:09.874216 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866083 2567 flags.go:64] FLAG: --v="2" Nov 01 06:09:09.874216 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866087 2567 flags.go:64] FLAG: --version="false" Nov 01 06:09:09.874216 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866092 2567 flags.go:64] FLAG: --vmodule="" Nov 01 06:09:09.874216 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866096 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 06:09:09.874216 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866100 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 06:09:09.874216 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866289 2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:09.874216 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866295 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:09.874216 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866299 2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:09.874216 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866301 2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:09.874216 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866305 2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:09.874216 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866308 2567 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:09.874216 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866311 2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:09.874216 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866313 2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:09.874216 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866316 2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:09.874216 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866319 2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:09.874216 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866322 2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:09.874216 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866325 2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:09.874216 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866328 2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:09.874216 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866331 2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:09.874918 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866333 2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:09.874918 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866336 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:09.874918 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866339 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:09.874918 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866342 2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:09.874918 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866344 2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:09.874918 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866347 2567 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:09.874918 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866350 2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:09.874918 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866354 2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:09.874918 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866356 2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:09.874918 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866359 2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:09.874918 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866362 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:09.874918 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866365 2567 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:09.874918 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866367 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:09.874918 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866370 2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:09.874918 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866372 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:09.874918 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866375 2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:09.874918 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866378 2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:09.874918 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866380 2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:09.874918 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866383 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:09.875430 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866386 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:09.875430 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866397 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:09.875430 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866400 2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:09.875430 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866402 2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:09.875430 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866405 2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:09.875430 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866407 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:09.875430 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866410 2567 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:09.875430 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866413 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:09.875430 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866415 2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:09.875430 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866418 2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:09.875430 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866421 2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:09.875430 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866424 2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:09.875430 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866426 2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:09.875430 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866429 2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:09.875430 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866431 2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:09.875430 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866434 2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:09.875430 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866437 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:09.875430 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866439 2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:09.875430 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866442 2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:09.875430 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866445 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:09.876082 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866449 2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:09.876082 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866451 2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:09.876082 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866454 2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:09.876082 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866458 2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:09.876082 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866462 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:09.876082 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866465 2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:09.876082 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866468 2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:09.876082 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866471 2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:09.876082 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866474 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:09.876082 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866476 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:09.876082 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866479 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:09.876082 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866481 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:09.876082 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866484 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:09.876082 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866487 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:09.876082 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866495 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:09.876082 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866498 2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:09.876082 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866501 2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:09.876082 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866504 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:09.876082 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866506 2567 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:09.876589 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866509 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:09.876589 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866511 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:09.876589 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866514 2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:09.876589 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866516 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:09.876589 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866519 2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:09.876589 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866522 2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:09.876589 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866525 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:09.876589 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866528 2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:09.876589 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866530 2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:09.876589 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866533 2567 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:09.876589 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866535 2567 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:09.876589 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866538 2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:09.876589 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866541 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:09.876589 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866547 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:09.876589 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866549 2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:09.876589 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866552 2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:09.876589 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866555 2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:09.876589 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866558 2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:09.876589 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866561 2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:09.877157 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866564 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:09.877157 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866567 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:09.877157 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866570 2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:09.877157 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866573 2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:09.877157 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866576 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:09.877157 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866578 2567 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:09.877157 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866581 2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:09.877157 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866584 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:09.877157 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866586 2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:09.877157 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866594 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:09.877157 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866597 2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:09.877157 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866600 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:09.877157 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866602 2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:09.877157 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866605 2567 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:09.877157 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866608 2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:09.877557 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.866636 2567 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 06:09:09.877557 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866688 2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:09.877557 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866693 2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:09.877557 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866696 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:09.877557 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866700 2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:09.877557 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866703 2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:09.877557 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866706 2567 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:09.877557 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866709 2567 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:09.877557 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866711 2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:09.877557 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866714 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:09.877557 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866716 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:09.877557 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866721 2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:09.877557 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866724 2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:09.877557 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866727 2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:09.877557 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866730 2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:09.877557 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866734 2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:09.878008 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866737 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:09.878008 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866740 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:09.878008 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866743 2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:09.878008 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866746 2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:09.878008 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866748 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:09.878008 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866751 2567 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:09.878008 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866754 2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:09.878008 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866757 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:09.878008 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866759 2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:09.878008 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866762 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:09.878008 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866764 2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:09.878008 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866772 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:09.878008 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866775 2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:09.878008 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866778 2567 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:09.878008 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866781 2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:09.878008 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866783 2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:09.878008 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866786 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:09.878008 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866789 2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:09.878008 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866792 2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:09.878506 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866796 2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:09.878506 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866805 2567 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:09.878506 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866808 2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:09.878506 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866811 2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:09.878506 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866814 2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:09.878506 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866816 2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:09.878506 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866819 2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:09.878506 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866821 2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:09.878506 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866826 2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:09.878506 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866844 2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:09.878506 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866848 2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:09.878506 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866850 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:09.878506 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866853 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:09.878506 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866856 2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:09.878506 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866858 2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:09.878506 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866861 2567 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:09.878506 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866864 2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:09.878506 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866866 2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:09.878506 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866869 2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:09.878506 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866872 2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:09.879021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866875 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:09.879021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866877 2567 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:09.879021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866880 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:09.879021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866882 2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:09.879021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866885 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:09.879021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866894 2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:09.879021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866896 2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:09.879021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866899 2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:09.879021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866902 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:09.879021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866904 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:09.879021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866907 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:09.879021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866909 2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:09.879021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866914 2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:09.879021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866917 2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:09.879021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866919 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:09.879021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866922 2567 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:09.879021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866925 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:09.879021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866927 2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:09.879021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866936 2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:09.879021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866939 2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:09.879505 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866943 2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:09.879505 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866946 2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:09.879505 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866948 2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:09.879505 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866951 2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:09.879505 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866953 2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:09.879505 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866956 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:09.879505 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866959 2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:09.879505 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866962 2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:09.879505 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866964 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:09.879505 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866967 2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:09.879505 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866969 2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:09.879505 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866972 2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:09.879505 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866976 2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:09.879505 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866980 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:09.879505 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866983 2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:09.879505 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866986 2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:09.879505 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866989 2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:09.879505 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.866991 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:09.879505 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.867000 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:09.879964 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.867003 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:09.879964 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.867006 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:09.879964 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.867008 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:09.879964 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.867011 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:09.879964 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.867013 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:09.879964 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.867017 2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:09.879964 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.867020 2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:09.879964 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.867023 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:09.879964 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.867025 2567 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:09.879964 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.867028 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:09.879964 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.867031 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:09.879964 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.867033 2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:09.879964 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.867036 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:09.879964 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.876181 2567 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 06:09:09.879964 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.876205 2567 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 06:09:09.879964 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.876223 2567 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 06:09:09.879964 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.876228 2567 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 06:09:09.879964 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876298 2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:09.879964 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876304 2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:09.880421 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876307 2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:09.880421 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876312 2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:09.880421 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876317 2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:09.880421 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876321 2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:09.880421 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876325 2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:09.880421 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876328 2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:09.880421 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876331 2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:09.880421 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876334 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:09.880421 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876337 2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:09.880421 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876340 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:09.880421 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876342 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:09.880421 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876345 2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:09.880421 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876348 2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:09.880421 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876351 2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:09.880421 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876353 2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:09.880421 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876356 2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:09.880421 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876359 2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:09.880421 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876361 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:09.880421 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876364 2567 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:09.880991 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876367 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:09.880991 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876369 2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:09.880991 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876372 2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:09.880991 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876375 2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:09.880991 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876377 2567 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:09.880991 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876380 2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:09.880991 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876383 2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:09.880991 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876386 2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:09.880991 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876390 2567 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:09.880991 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876393 2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:09.880991 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876396 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:09.880991 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876399 2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:09.880991 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876401 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:09.880991 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876404 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:09.880991 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876407 2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:09.880991 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876409 2567 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:09.880991 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876411 2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:09.880991 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876415 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:09.880991 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876418 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:09.880991 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876420 2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:09.881451 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876423 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:09.881451 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876427 2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:09.881451 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876431 2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:09.881451 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876435 2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:09.881451 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876438 2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:09.881451 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876447 2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:09.881451 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876450 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:09.881451 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876453 2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:09.881451 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876456 2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:09.881451 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876459 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:09.881451 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876462 2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:09.881451 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876465 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:09.881451 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876467 2567 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:09.881451 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876470 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:09.881451 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876473 2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:09.881451 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876475 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:09.881451 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876478 2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:09.881451 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876480 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:09.881451 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876484 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:09.881928 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876487 2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:09.881928 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876490 2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:09.881928 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876492 2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:09.881928 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876495 2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:09.881928 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876498 2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:09.881928 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876501 2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:09.881928 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876503 2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:09.881928 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876506 2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:09.881928 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876509 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:09.881928 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876512 2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:09.881928 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876514 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:09.881928 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876517 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:09.881928 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876520 2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:09.881928 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876522 2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:09.881928 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876525 2567 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:09.881928 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876528 2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:09.881928 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876531 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:09.881928 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876533 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:09.881928 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876536 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:09.881928 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876545 2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:09.882466 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876548 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:09.882466 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876550 2567 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:09.882466 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876553 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:09.882466 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876556 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:09.882466 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876558 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:09.882466 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876561 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:09.882466 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876563 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:09.882466 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876566 2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:09.882466 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876569 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:09.882466 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876571 2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:09.882466 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876574 2567 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:09.882466 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876577 2567 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:09.882466 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876580 2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:09.882466 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876584 2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:09.882466 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876586 2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:09.882466 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876589 2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:09.882466 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876592 2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:09.882466 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876595 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:09.882466 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876597 2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:09.882466 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876601 2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:09.882957 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876604 2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:09.882957 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876606 2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:09.882957 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876609 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:09.882957 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876612 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:09.882957 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876614 2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:09.882957 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876617 2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:09.882957 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.876650 2567 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 06:09:09.882957 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876801 2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:09.882957 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876807 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:09.882957 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876810 2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:09.882957 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876814 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:09.882957 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876818 2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:09.882957 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876826 2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:09.882957 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876847 2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:09.882957 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876851 2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:09.882957 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876854 2567 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:09.883323 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876856 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:09.883323 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876859 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:09.883323 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876862 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:09.883323 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876865 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:09.883323 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876868 2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:09.883323 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876872 2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:09.883323 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876876 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:09.883323 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876879 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:09.883323 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876882 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:09.883323 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876885 2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:09.883323 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876888 2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:09.883323 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876891 2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:09.883323 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876893 2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:09.883323 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876896 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:09.883323 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876899 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:09.883323 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876901 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:09.883323 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876904 2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:09.883323 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876907 2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:09.883323 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876910 2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:09.883773 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876912 2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:09.883773 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876915 2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:09.883773 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876917 2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:09.883773 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876920 2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:09.883773 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876923 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:09.883773 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876925 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:09.883773 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876928 2567 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:09.883773 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876930 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:09.883773 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876933 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:09.883773 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876936 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:09.883773 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876940 2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:09.883773 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876943 2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:09.883773 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876945 2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:09.883773 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876948 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:09.883773 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876950 2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:09.883773 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876953 2567 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:09.883773 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876956 2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:09.883773 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876958 2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:09.883773 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876961 2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:09.883773 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876964 2567 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:09.884259 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876966 2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:09.884259 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876969 2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:09.884259 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876972 2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:09.884259 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876975 2567 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:09.884259 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876978 2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:09.884259 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876981 2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:09.884259 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876983 2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:09.884259 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876986 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:09.884259 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876989 2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:09.884259 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876992 2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:09.884259 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876994 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:09.884259 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876997 2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:09.884259 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.876999 2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:09.884259 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877002 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:09.884259 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877005 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:09.884259 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877007 2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:09.884259 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877010 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:09.884259 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877012 2567 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:09.884259 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877015 2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:09.884259 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877018 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:09.884736 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877020 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:09.884736 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877023 2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:09.884736 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877025 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:09.884736 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877029 2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:09.884736 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877031 2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:09.884736 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877034 2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:09.884736 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877036 2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:09.884736 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877039 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:09.884736 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877041 2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:09.884736 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877044 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:09.884736 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877046 2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:09.884736 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877049 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:09.884736 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877052 2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:09.884736 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877054 2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:09.884736 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877058 2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:09.884736 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877060 2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:09.884736 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877063 2567 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:09.884736 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877066 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:09.884736 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877068 2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:09.885191 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877071 2567 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:09.885191 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877074 2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:09.885191 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877076 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:09.885191 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877079 2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:09.885191 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877082 2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:09.885191 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877084 2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:09.885191 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877087 2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:09.885191 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877089 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:09.885191 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877092 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:09.885191 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877095 2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:09.885191 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877097 2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:09.885191 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877100 2567 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:09.885191 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877103 2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:09.885191 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877105 2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:09.885191 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877108 2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:09.885191 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877111 2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:09.885191 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877113 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:09.885191 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877117 2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:09.885191 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877119 2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:09.885629 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.877145 2567 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 06:09:09.885629 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877201 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:09.885629 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877206 2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:09.885629 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877209 2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:09.885629 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877212 2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:09.885629 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877215 2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:09.885629 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877218 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:09.885629 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877221 2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:09.885629 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877224 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:09.885629 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877227 2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:09.885629 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877230 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:09.885629 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877232 2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:09.885629 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877235 2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:09.885629 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877237 2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:09.885629 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877240 2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:09.886021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877243 2567 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:09.886021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877245 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:09.886021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877248 2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:09.886021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877251 2567 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:09.886021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877254 2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:09.886021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877257 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:09.886021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877259 2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:09.886021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877262 2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:09.886021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877265 2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:09.886021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877267 2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:09.886021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877270 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:09.886021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877273 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:09.886021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877275 2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:09.886021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877278 2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:09.886021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877281 2567 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:09.886021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877283 2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:09.886021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877286 2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:09.886021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877289 2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:09.886021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877292 2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:09.886021 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877295 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:09.886567 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877297 2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:09.886567 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877300 2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:09.886567 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877304 2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:09.886567 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877307 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:09.886567 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877311 2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:09.886567 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877314 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:09.886567 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877317 2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:09.886567 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877319 2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:09.886567 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877322 2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:09.886567 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877325 2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:09.886567 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877327 2567 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:09.886567 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877330 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:09.886567 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877332 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:09.886567 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877335 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:09.886567 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877337 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:09.886567 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877340 2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:09.886567 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877344 2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:09.886567 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877348 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:09.886567 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877350 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:09.887062 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877353 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:09.887062 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877356 2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:09.887062 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877359 2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:09.887062 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877364 2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:09.887062 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877366 2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:09.887062 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877369 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:09.887062 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877372 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:09.887062 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877374 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:09.887062 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877377 2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:09.887062 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877380 2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:09.887062 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877383 2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:09.887062 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877385 2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:09.887062 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877388 2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:09.887062 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877391 2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:09.887062 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877393 2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:09.887062 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877396 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:09.887062 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877398 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:09.887062 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877401 2567 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:09.887062 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877404 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:09.887497 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877407 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:09.887497 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877409 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:09.887497 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877412 2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:09.887497 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877415 2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:09.887497 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877417 2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:09.887497 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877420 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:09.887497 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877422 2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:09.887497 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877425 2567 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:09.887497 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877428 2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:09.887497 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877431 2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:09.887497 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877434 2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:09.887497 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877436 2567 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:09.887497 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877439 2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:09.887497 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877442 2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:09.887497 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877444 2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:09.887497 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877447 2567 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:09.887497 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877454 2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:09.887497 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877457 2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:09.887497 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877460 2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:09.887497 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877463 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:09.887979 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877465 2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:09.887979 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877468 2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:09.887979 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877471 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:09.887979 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877474 2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:09.887979 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877477 2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:09.887979 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877480 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:09.887979 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877482 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:09.887979 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877485 2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:09.887979 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877488 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:09.887979 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877490 2567 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:09.887979 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877494 2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:09.887979 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877500 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:09.887979 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877503 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:09.887979 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:09.877506 2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:09.887979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.877574 2567 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 06:09:09.887979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.879566 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 06:09:09.887979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.880380 2567 server.go:1014] "Starting client certificate rotation" Nov 01 06:09:09.887979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.880476 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 06:09:09.888403 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.880520 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 06:09:09.888403 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.882033 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 06:09:09.888403 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.886315 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 06:09:09.897371 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.897349 2567 log.go:25] "Validated CRI v1 runtime API" Nov 01 06:09:09.901613 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.901595 2567 log.go:25] "Validated CRI v1 image API" Nov 01 06:09:09.903185 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.903162 2567 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 06:09:09.906408 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.906379 2567 fs.go:135] Filesystem UUIDs: map[1fb5b39b-261d-4c25-b15e-9a12c21bee9f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f19ef59e-80e2-4476-a60f-7c1f7b777a0d:/dev/nvme0n1p4] Nov 01 06:09:09.906477 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.906407 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 06:09:09.913859 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.913552 2567 manager.go:217] Machine: {Timestamp:2026-11-01 06:09:09.911638107 +0000 UTC m=+0.301823513 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100585 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c53dd4d080a6e06eb0162035b0773 SystemUUID:ec2c53dd-4d08-0a6e-06eb-0162035b0773 BootID:6cc7d41c-5410-4001-8366-8ead92311cc5 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:52:0a:10:b7:43 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:52:0a:10:b7:43 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:c8:8a:2e:4a:c6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 06:09:09.913997 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.913984 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 06:09:09.914174 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.914150 2567 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 06:09:09.915107 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.915081 2567 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 06:09:09.915245 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.915110 2567 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-68.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 06:09:09.915292 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.915248 2567 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 06:09:09.915292 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.915257 2567 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 06:09:09.915348 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.915295 2567 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 06:09:09.915348 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.915314 2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 06:09:09.915945 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.915935 2567 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 06:09:09.919520 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.919504 2567 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 06:09:09.919657 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.919647 2567 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 06:09:09.920315 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.920304 2567 kubelet.go:493] "Attempting to sync node with API server" Nov 01 06:09:09.920345 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.920321 2567 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 06:09:09.920345 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.920334 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 06:09:09.920345 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.920344 2567 kubelet.go:405] "Adding apiserver pod source" Nov 01 06:09:09.920437 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.920353 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 06:09:09.920729 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.920713 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 06:09:09.922642 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.922624 2567 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 06:09:09.927005 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.926987 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 06:09:09.928142 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.928127 2567 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 06:09:09.928189 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.928159 2567 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 06:09:09.928346 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.928335 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 06:09:09.928380 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.928351 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 06:09:09.928380 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.928364 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 06:09:09.928380 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.928371 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 06:09:09.928380 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.928377 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 06:09:09.928523 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.928382 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 06:09:09.928753 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.928745 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 06:09:09.928784 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.928754 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 06:09:09.928784 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.928763 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 06:09:09.928784 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.928769 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 06:09:09.928784 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.928781 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 06:09:09.928976 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.928791 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 06:09:09.929292 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.929282 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 06:09:09.929292 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.929292 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 06:09:09.933136 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.933123 2567 server.go:1263] "Started kubelet" Nov 01 06:09:09.933235 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.933203 2567 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 06:09:09.933288 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.933208 2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 06:09:09.933330 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.933294 2567 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 06:09:09.934015 ip-10-0-140-68 systemd[1]: Started Kubernetes Kubelet. Nov 01 06:09:09.934369 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.934348 2567 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 06:09:09.936191 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.936173 2567 server.go:317] "Adding debug handlers to kubelet server" Nov 01 06:09:09.937969 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:09.937936 2567 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 06:09:09.941095 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.941078 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 06:09:09.941904 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.941887 2567 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 06:09:09.941998 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:09.941981 2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-68.us-west-1.compute.internal\" not found" Nov 01 06:09:09.942360 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.942346 2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 06:09:09.942360 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.942349 2567 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 06:09:09.942485 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.942371 2567 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 06:09:09.942598 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.942489 2567 reconstruct.go:97] "Volume reconstruction finished" Nov 01 06:09:09.942598 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.942497 2567 reconciler.go:29] "Reconciler: start to sync state" Nov 01 06:09:09.942743 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.942727 2567 factory.go:55] Registering systemd factory Nov 01 06:09:09.942779 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.942748 2567 factory.go:223] Registration of the systemd container factory successfully Nov 01 06:09:09.942975 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.942962 2567 factory.go:153] Registering CRI-O factory Nov 01 06:09:09.943020 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.942977 2567 factory.go:223] Registration of the crio container factory successfully Nov 01 06:09:09.943056 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.943046 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 06:09:09.943091 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.943064 2567 factory.go:103] Registering Raw factory Nov 01 06:09:09.943091 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.943076 2567 manager.go:1199] Started watching for new ooms in manager Nov 01 06:09:09.943749 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.943738 2567 manager.go:319] Starting recovery of all containers Nov 01 06:09:09.956870 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.956806 2567 manager.go:324] Recovery completed Nov 01 06:09:09.961364 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.961346 2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:09.963135 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.963105 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:09.963245 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.963154 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:09.963245 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.963169 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:09.963756 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.963743 2567 cpu_manager.go:226] "Starting" policy="none" Nov 01 06:09:09.963756 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.963753 2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:09.963926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.963754 2567 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 06:09:09.963926 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.963819 2567 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 06:09:09.964523 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.964509 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:09.964581 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.964537 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:09.964581 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.964547 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:09.965963 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.965951 2567 policy_none.go:50] "Start" Nov 01 06:09:09.966010 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.965968 2567 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 06:09:09.966010 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.965978 2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 06:09:09.967922 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:09.967910 2567 policy_none.go:44] "Start" Nov 01 06:09:10.001780 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.001543 2567 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 06:09:10.005956 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.005939 2567 manager.go:341] "Starting Device Plugin manager" Nov 01 06:09:10.006075 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:10.005980 2567 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 06:09:10.006075 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.005993 2567 server.go:87] "Starting device plugin registration server" Nov 01 06:09:10.006364 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.006350 2567 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 06:09:10.006413 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.006367 2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 06:09:10.006498 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.006468 2567 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 06:09:10.006613 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.006562 2567 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 06:09:10.006613 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.006568 2567 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 06:09:10.007183 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:10.007161 2567 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 06:09:10.007269 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:10.007220 2567 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-68.us-west-1.compute.internal\" not found" Nov 01 06:09:10.075664 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.075622 2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-68.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 01 06:09:10.077738 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:10.075622 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-68.us-west-1.compute.internal.18e3d6e3aaaed081 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-68.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-68.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 06:09:09.933092993 +0000 UTC m=+0.323278398,LastTimestamp:2026-11-01 06:09:09.933092993 +0000 UTC m=+0.323278398,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-68.us-west-1.compute.internal,}" Nov 01 06:09:10.080125 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.080098 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 06:09:10.081415 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.081392 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 06:09:10.081415 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.081416 2567 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 06:09:10.081555 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.081434 2567 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 06:09:10.081555 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:10.081467 2567 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 06:09:10.107212 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.107116 2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:10.108202 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.108183 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:10.108310 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.108221 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:10.108310 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.108237 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:10.108310 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.108269 2567 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:10.124038 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:10.123999 2567 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-68.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 01 06:09:10.152942 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.152912 2567 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 06:09:10.153090 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.152963 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7qf9v" Nov 01 06:09:10.182678 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.182639 2567 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal"] Nov 01 06:09:10.182824 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.182711 2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:10.183693 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.183678 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:10.183750 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.183711 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:10.183750 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.183721 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:10.186038 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.186025 2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:10.186196 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.186181 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:10.186232 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.186223 2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:10.186812 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.186797 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:10.186917 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.186825 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:10.186917 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.186800 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:10.186917 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.186863 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:10.186917 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.186869 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:10.186917 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.186903 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:10.189088 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.189072 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:10.189186 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.189106 2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:10.190151 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.190135 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:10.190221 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.190167 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:10.190221 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.190185 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:10.220857 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:10.220813 2567 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-68.us-west-1.compute.internal\" not found" node="ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:10.231720 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.231673 2567 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 06:09:10.232607 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.232581 2567 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 06:09:10.236192 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:10.236166 2567 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-68.us-west-1.compute.internal\" not found" node="ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:10.243883 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.243828 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/645c51de28f312aaa75d7c297b03488b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal\" (UID: \"645c51de28f312aaa75d7c297b03488b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:10.243977 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.243900 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/58599ac63e6a6b0b2eb7aede2e6e233a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal\" (UID: \"58599ac63e6a6b0b2eb7aede2e6e233a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:10.244032 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.243995 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58599ac63e6a6b0b2eb7aede2e6e233a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal\" (UID: \"58599ac63e6a6b0b2eb7aede2e6e233a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:10.329129 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.329092 2567 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 06:09:10.330794 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.330777 2567 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 06:09:10.333950 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.333930 2567 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:10.334022 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:10.333959 2567 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-68.us-west-1.compute.internal\": node \"ip-10-0-140-68.us-west-1.compute.internal\" not found" Nov 01 06:09:10.342307 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.342286 2567 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:10.344460 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.344442 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/645c51de28f312aaa75d7c297b03488b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal\" (UID: \"645c51de28f312aaa75d7c297b03488b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:10.344504 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.344478 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/58599ac63e6a6b0b2eb7aede2e6e233a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal\" (UID: \"58599ac63e6a6b0b2eb7aede2e6e233a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:10.344537 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.344522 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58599ac63e6a6b0b2eb7aede2e6e233a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal\" (UID: \"58599ac63e6a6b0b2eb7aede2e6e233a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:10.344569 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.344558 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/58599ac63e6a6b0b2eb7aede2e6e233a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal\" (UID: \"58599ac63e6a6b0b2eb7aede2e6e233a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:10.344600 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.344566 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58599ac63e6a6b0b2eb7aede2e6e233a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal\" (UID: \"58599ac63e6a6b0b2eb7aede2e6e233a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:10.344637 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.344616 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/645c51de28f312aaa75d7c297b03488b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal\" (UID: \"645c51de28f312aaa75d7c297b03488b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:10.402074 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.401999 2567 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 06:09:10.402198 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.402119 2567 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:10.455989 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.455955 2567 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 06:09:10.523684 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.523646 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:10.540300 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.540270 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:10.880304 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.880207 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 06:09:10.880872 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.880421 2567 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 06:09:10.880872 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.880451 2567 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 06:09:10.880872 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.880457 2567 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 06:09:10.880872 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:10.880459 2567 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-140-68.us-west-1.compute.internal.18e3d6e3ac79d700\": read tcp 10.0.140.68:43684->3.140.212.52:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-140-68.us-west-1.compute.internal.18e3d6e3ac79d700 default 5131 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-68.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-68.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-68.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 06:09:09 +0000 UTC,LastTimestamp:2026-11-01 06:09:10.108241995 +0000 UTC m=+0.498427402,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-68.us-west-1.compute.internal,}" Nov 01 06:09:10.921300 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.921270 2567 apiserver.go:52] "Watching apiserver" Nov 01 06:09:10.941252 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:10.941217 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 06:09:11.158316 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.158190 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 06:04:10 +0000 UTC" deadline="2028-04-27 06:55:04.806941065 +0000 UTC" Nov 01 06:09:11.158316 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.158258 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13032h45m53.648686687s" Nov 01 06:09:11.183956 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.183925 2567 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 06:09:11.184291 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.184272 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb","openshift-multus/multus-sxts6","openshift-multus/network-metrics-daemon-b98x2","kube-system/global-pull-secret-syncer-txvqp","kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal","kube-system/konnectivity-agent-h8j7v","openshift-dns/node-resolver-ks27g","openshift-image-registry/node-ca-vth7q","openshift-multus/multus-additional-cni-plugins-fvnw6","openshift-network-diagnostics/network-check-target-vvpd2","openshift-cluster-node-tuning-operator/tuned-258pf","openshift-network-operator/iptables-alerter-rb97g","openshift-ovn-kubernetes/ovnkube-node-wj2f6"] Nov 01 06:09:11.186671 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.186655 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.188801 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.188778 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.190768 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.190753 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:11.190857 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:11.190811 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b98x2" podUID="8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8" Nov 01 06:09:11.192757 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.192737 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:11.192855 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:11.192810 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txvqp" podUID="5b0855c2-93a3-44a5-9311-63d891705e0e" Nov 01 06:09:11.194816 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.194799 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h8j7v" Nov 01 06:09:11.196943 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.196926 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ks27g" Nov 01 06:09:11.199184 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.199168 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vth7q" Nov 01 06:09:11.201720 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.201336 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.204552 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.204525 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:11.204668 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:11.204609 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvpd2" podUID="6228c388-659f-4ca7-8d51-4fa5658424e2" Nov 01 06:09:11.206651 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.206633 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.208651 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.208634 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rb97g" Nov 01 06:09:11.210847 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.210816 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.238431 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.238393 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 06:09:11.238600 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.238575 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 06:09:11.238600 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.238572 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 06:09:11.238719 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.238621 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-78q7p\"" Nov 01 06:09:11.240065 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.240043 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 06:09:11.240197 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.240126 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 06:09:11.240197 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.240136 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rnlbn\"" Nov 01 06:09:11.240306 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.240197 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 06:09:11.240306 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.240246 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 06:09:11.244166 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.244143 2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 06:09:11.246054 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.246034 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 06:09:11.246140 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.246066 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 06:09:11.246140 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.246081 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vsdrw\"" Nov 01 06:09:11.248039 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.248024 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 06:09:11.248099 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.248069 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hnk2x\"" Nov 01 06:09:11.248099 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.248086 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 06:09:11.249174 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249157 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-conf-dir\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.249223 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249182 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:11.249223 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249200 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhqdn\" (UniqueName: \"kubernetes.io/projected/3b623310-9483-47a2-a5de-f19b87819eda-kube-api-access-rhqdn\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.249316 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249224 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-var-lib-kubelet\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.249316 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249270 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/199a6bc2-3668-415c-8cc4-813f96eccfa4-agent-certs\") pod \"konnectivity-agent-h8j7v\" (UID: \"199a6bc2-3668-415c-8cc4-813f96eccfa4\") " pod="kube-system/konnectivity-agent-h8j7v" Nov 01 06:09:11.249394 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249317 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr4ql\" (UniqueName: \"kubernetes.io/projected/93f3367a-3295-4f1c-931a-ba9c643d3b76-kube-api-access-xr4ql\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.249394 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249352 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-var-lib-openvswitch\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.249394 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249381 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b0855c2-93a3-44a5-9311-63d891705e0e-dbus\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:11.249505 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249401 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b856c58-9b9e-47b3-947a-ed56ae892e9b-serviceca\") pod \"node-ca-vth7q\" (UID: \"7b856c58-9b9e-47b3-947a-ed56ae892e9b\") " pod="openshift-image-registry/node-ca-vth7q" Nov 01 06:09:11.249505 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249438 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt9qk\" (UniqueName: \"kubernetes.io/projected/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-kube-api-access-qt9qk\") pod \"network-metrics-daemon-b98x2\" (UID: \"8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8\") " pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:11.249505 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249457 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-etc-selinux\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.249505 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249472 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/199a6bc2-3668-415c-8cc4-813f96eccfa4-konnectivity-ca\") pod \"konnectivity-agent-h8j7v\" (UID: \"199a6bc2-3668-415c-8cc4-813f96eccfa4\") " pod="kube-system/konnectivity-agent-h8j7v" Nov 01 06:09:11.249505 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249489 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/939039b6-e071-4a27-b24e-20bb2669dc15-tmp-dir\") pod \"node-resolver-ks27g\" (UID: \"939039b6-e071-4a27-b24e-20bb2669dc15\") " pod="openshift-dns/node-resolver-ks27g" Nov 01 06:09:11.249708 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249522 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-sysctl-conf\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.249708 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249579 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-run-openvswitch\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.249708 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249618 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/24ddad96-0a15-4c10-b302-416833e87746-ovnkube-config\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.249708 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249655 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-socket-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.249708 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249685 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-sys-fs\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.249949 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249710 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-cni-dir\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.249949 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249737 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-systemd\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.249949 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249804 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/24ddad96-0a15-4c10-b302-416833e87746-env-overrides\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.249949 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249854 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b623310-9483-47a2-a5de-f19b87819eda-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.249949 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249881 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-sys\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.249949 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249906 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-var-lib-kubelet\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.249949 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249929 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb7kp\" (UniqueName: \"kubernetes.io/projected/b490df30-7bfc-4c25-a9f5-768733b3b6e6-kube-api-access-rb7kp\") pod \"iptables-alerter-rb97g\" (UID: \"b490df30-7bfc-4c25-a9f5-768733b3b6e6\") " pod="openshift-network-operator/iptables-alerter-rb97g" Nov 01 06:09:11.250259 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249951 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-run-systemd\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.250259 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.249975 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-cni-bin\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.250259 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250000 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-cni-netd\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.250259 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250033 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-registration-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.250259 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250056 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-socket-dir-parent\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.250259 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250086 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b623310-9483-47a2-a5de-f19b87819eda-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.250259 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250116 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-modprobe-d\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.250259 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250162 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-kubernetes\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.250259 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250187 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-host\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.250259 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250209 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-run-netns\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.250641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250270 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-run-multus-certs\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.250641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250294 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z2gw\" (UniqueName: \"kubernetes.io/projected/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-kube-api-access-4z2gw\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.250641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250309 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd7r9\" (UniqueName: \"kubernetes.io/projected/939039b6-e071-4a27-b24e-20bb2669dc15-kube-api-access-vd7r9\") pod \"node-resolver-ks27g\" (UID: \"939039b6-e071-4a27-b24e-20bb2669dc15\") " pod="openshift-dns/node-resolver-ks27g" Nov 01 06:09:11.250641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250323 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-run-ovn\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.250641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250350 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-log-socket\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.250641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250393 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b623310-9483-47a2-a5de-f19b87819eda-system-cni-dir\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.250641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250431 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wqjh\" (UniqueName: \"kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh\") pod \"network-check-target-vvpd2\" (UID: \"6228c388-659f-4ca7-8d51-4fa5658424e2\") " pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:11.250641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250468 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b490df30-7bfc-4c25-a9f5-768733b3b6e6-iptables-alerter-script\") pod \"iptables-alerter-rb97g\" (UID: \"b490df30-7bfc-4c25-a9f5-768733b3b6e6\") " pod="openshift-network-operator/iptables-alerter-rb97g" Nov 01 06:09:11.250641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250502 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-run-ovn-kubernetes\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.250641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250508 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 06:09:11.250641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250520 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 06:09:11.250641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250523 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vcgd\" (UniqueName: \"kubernetes.io/projected/24ddad96-0a15-4c10-b302-416833e87746-kube-api-access-4vcgd\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.250641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250537 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 06:09:11.250641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250568 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-sysctl-d\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.250641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250601 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-system-cni-dir\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.250641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250632 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-daemon-config\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.251151 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250650 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b0855c2-93a3-44a5-9311-63d891705e0e-kubelet-config\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:11.251151 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250683 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njlqm\" (UniqueName: \"kubernetes.io/projected/7b856c58-9b9e-47b3-947a-ed56ae892e9b-kube-api-access-njlqm\") pod \"node-ca-vth7q\" (UID: \"7b856c58-9b9e-47b3-947a-ed56ae892e9b\") " pod="openshift-image-registry/node-ca-vth7q" Nov 01 06:09:11.251151 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250710 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/939039b6-e071-4a27-b24e-20bb2669dc15-hosts-file\") pod \"node-resolver-ks27g\" (UID: \"939039b6-e071-4a27-b24e-20bb2669dc15\") " pod="openshift-dns/node-resolver-ks27g" Nov 01 06:09:11.251151 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250725 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-lib-modules\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.251151 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250786 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-device-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.251151 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250805 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-run-k8s-cni-cncf-io\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.251151 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250822 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b856c58-9b9e-47b3-947a-ed56ae892e9b-host\") pod \"node-ca-vth7q\" (UID: \"7b856c58-9b9e-47b3-947a-ed56ae892e9b\") " pod="openshift-image-registry/node-ca-vth7q" Nov 01 06:09:11.251151 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250854 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b490df30-7bfc-4c25-a9f5-768733b3b6e6-host-slash\") pod \"iptables-alerter-rb97g\" (UID: \"b490df30-7bfc-4c25-a9f5-768733b3b6e6\") " pod="openshift-network-operator/iptables-alerter-rb97g" Nov 01 06:09:11.251151 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250888 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-slash\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.251151 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250916 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-var-lib-cni-bin\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.251151 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250949 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b623310-9483-47a2-a5de-f19b87819eda-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.251151 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250974 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs\") pod \"network-metrics-daemon-b98x2\" (UID: \"8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8\") " pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:11.251151 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.250992 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-node-log\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.251151 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.251023 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tjfc\" (UniqueName: \"kubernetes.io/projected/d48e417b-1b11-4ee3-8b45-0a71f68459bb-kube-api-access-4tjfc\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.251151 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.251056 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-kubelet\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.251151 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.251094 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-systemd-units\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.251771 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.251111 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/24ddad96-0a15-4c10-b302-416833e87746-ovnkube-script-lib\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.251771 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.251126 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b623310-9483-47a2-a5de-f19b87819eda-cnibin\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.251771 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.251144 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b623310-9483-47a2-a5de-f19b87819eda-cni-binary-copy\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.251771 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.251180 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-sysconfig\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.251771 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.251210 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-run\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.251771 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.251228 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-tuned\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.251771 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.251274 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-etc-openvswitch\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.251771 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.251303 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/24ddad96-0a15-4c10-b302-416833e87746-ovn-node-metrics-cert\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.251771 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.251327 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-etc-kubernetes\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.251771 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.251350 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.251771 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.251385 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-cnibin\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.251771 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.251408 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-hostroot\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.251771 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.251431 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93f3367a-3295-4f1c-931a-ba9c643d3b76-tmp\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.251771 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.251458 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.251771 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.251482 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d48e417b-1b11-4ee3-8b45-0a71f68459bb-cni-binary-copy\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.251771 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.251519 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-run-netns\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.251771 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.251542 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-var-lib-cni-multus\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.335968 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.335947 2567 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 06:09:11.336070 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.336008 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-z4rjv" Nov 01 06:09:11.339226 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.339201 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 06:09:11.339226 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.339228 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 06:09:11.339542 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.339265 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 06:09:11.339542 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.339280 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sdp5j\"" Nov 01 06:09:11.339542 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.339300 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wss87\"" Nov 01 06:09:11.339542 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.339407 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 06:09:11.339542 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.339468 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 06:09:11.339542 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.339470 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 06:09:11.339542 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.339482 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 06:09:11.339542 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.339517 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 06:09:11.340767 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.340754 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 06:09:11.340767 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.340755 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 06:09:11.350799 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.350756 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6m98z\"" Nov 01 06:09:11.350799 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.350773 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 06:09:11.350799 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.350759 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 06:09:11.350799 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.350759 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vwtj8\"" Nov 01 06:09:11.351055 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.350789 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bnjvc\"" Nov 01 06:09:11.351055 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.350882 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 06:09:11.351980 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.351953 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-sysctl-d\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.351980 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.351975 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-system-cni-dir\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.352092 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.351991 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-daemon-config\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.352092 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352006 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b0855c2-93a3-44a5-9311-63d891705e0e-kubelet-config\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:11.352092 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352021 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njlqm\" (UniqueName: \"kubernetes.io/projected/7b856c58-9b9e-47b3-947a-ed56ae892e9b-kube-api-access-njlqm\") pod \"node-ca-vth7q\" (UID: \"7b856c58-9b9e-47b3-947a-ed56ae892e9b\") " pod="openshift-image-registry/node-ca-vth7q" Nov 01 06:09:11.352092 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352034 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/939039b6-e071-4a27-b24e-20bb2669dc15-hosts-file\") pod \"node-resolver-ks27g\" (UID: \"939039b6-e071-4a27-b24e-20bb2669dc15\") " pod="openshift-dns/node-resolver-ks27g" Nov 01 06:09:11.352092 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352048 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-lib-modules\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.352092 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352066 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-device-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.352092 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352087 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-system-cni-dir\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.352384 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352093 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b0855c2-93a3-44a5-9311-63d891705e0e-kubelet-config\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:11.352384 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352123 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-device-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.352384 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352125 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/939039b6-e071-4a27-b24e-20bb2669dc15-hosts-file\") pod \"node-resolver-ks27g\" (UID: \"939039b6-e071-4a27-b24e-20bb2669dc15\") " pod="openshift-dns/node-resolver-ks27g" Nov 01 06:09:11.352384 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352159 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-run-k8s-cni-cncf-io\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.352384 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352168 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-sysctl-d\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.352384 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352182 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b856c58-9b9e-47b3-947a-ed56ae892e9b-host\") pod \"node-ca-vth7q\" (UID: \"7b856c58-9b9e-47b3-947a-ed56ae892e9b\") " pod="openshift-image-registry/node-ca-vth7q" Nov 01 06:09:11.352384 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352199 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b490df30-7bfc-4c25-a9f5-768733b3b6e6-host-slash\") pod \"iptables-alerter-rb97g\" (UID: \"b490df30-7bfc-4c25-a9f5-768733b3b6e6\") " pod="openshift-network-operator/iptables-alerter-rb97g" Nov 01 06:09:11.352384 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352202 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-lib-modules\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.352384 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352219 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-run-k8s-cni-cncf-io\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.352384 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352221 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-slash\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.352384 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352256 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-slash\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.352384 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352260 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b490df30-7bfc-4c25-a9f5-768733b3b6e6-host-slash\") pod \"iptables-alerter-rb97g\" (UID: \"b490df30-7bfc-4c25-a9f5-768733b3b6e6\") " pod="openshift-network-operator/iptables-alerter-rb97g" Nov 01 06:09:11.352384 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352267 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-var-lib-cni-bin\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.352384 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352292 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b623310-9483-47a2-a5de-f19b87819eda-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.352384 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352307 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b856c58-9b9e-47b3-947a-ed56ae892e9b-host\") pod \"node-ca-vth7q\" (UID: \"7b856c58-9b9e-47b3-947a-ed56ae892e9b\") " pod="openshift-image-registry/node-ca-vth7q" Nov 01 06:09:11.352384 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352316 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs\") pod \"network-metrics-daemon-b98x2\" (UID: \"8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8\") " pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:11.352384 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352338 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-node-log\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.352384 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352344 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-var-lib-cni-bin\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.353181 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352374 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tjfc\" (UniqueName: \"kubernetes.io/projected/d48e417b-1b11-4ee3-8b45-0a71f68459bb-kube-api-access-4tjfc\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.353181 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352402 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-kubelet\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.353181 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352404 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-node-log\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.353181 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:11.352420 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:11.353181 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352425 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-systemd-units\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.353181 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352446 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/24ddad96-0a15-4c10-b302-416833e87746-ovnkube-script-lib\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.353181 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352455 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-kubelet\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.353181 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:11.352495 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs podName:8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:11.852454707 +0000 UTC m=+2.242640126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs") pod "network-metrics-daemon-b98x2" (UID: "8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:11.353181 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352498 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-systemd-units\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.353181 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352532 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b623310-9483-47a2-a5de-f19b87819eda-cnibin\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.353181 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352552 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b623310-9483-47a2-a5de-f19b87819eda-cni-binary-copy\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.353181 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352569 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-sysconfig\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.353181 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352586 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-run\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.353181 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352592 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-daemon-config\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.353181 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352631 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-tuned\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.353181 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352655 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-sysconfig\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.353181 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352656 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-etc-openvswitch\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.354037 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352693 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-etc-openvswitch\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.354037 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352697 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/24ddad96-0a15-4c10-b302-416833e87746-ovn-node-metrics-cert\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.354037 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352705 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b623310-9483-47a2-a5de-f19b87819eda-cnibin\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.354037 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352724 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-etc-kubernetes\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.354037 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352750 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.354037 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352775 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-cnibin\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.354037 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352799 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-hostroot\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.354037 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352807 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-etc-kubernetes\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.354037 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352822 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93f3367a-3295-4f1c-931a-ba9c643d3b76-tmp\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.354037 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352912 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b623310-9483-47a2-a5de-f19b87819eda-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.354037 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.352987 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-cnibin\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.354037 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353022 2567 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 06:09:11.354037 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353033 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.354037 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353046 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/24ddad96-0a15-4c10-b302-416833e87746-ovnkube-script-lib\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.354037 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353066 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-hostroot\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.354037 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353084 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.354037 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353137 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d48e417b-1b11-4ee3-8b45-0a71f68459bb-cni-binary-copy\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.354037 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353162 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-run-netns\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.355084 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353185 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-var-lib-cni-multus\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.355084 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353184 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.355084 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353185 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b623310-9483-47a2-a5de-f19b87819eda-cni-binary-copy\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.355084 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353225 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-conf-dir\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.355084 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353248 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-run\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.355084 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353258 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-var-lib-cni-multus\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.355084 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353253 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:11.355084 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353297 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-run-netns\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.355084 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353300 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqdn\" (UniqueName: \"kubernetes.io/projected/3b623310-9483-47a2-a5de-f19b87819eda-kube-api-access-rhqdn\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.355084 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:11.353315 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:11.355084 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353329 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-conf-dir\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.355084 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353332 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-var-lib-kubelet\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.355084 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:11.353361 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret podName:5b0855c2-93a3-44a5-9311-63d891705e0e nodeName:}" failed. No retries permitted until 2026-11-01 06:09:11.853345049 +0000 UTC m=+2.243530456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret") pod "global-pull-secret-syncer-txvqp" (UID: "5b0855c2-93a3-44a5-9311-63d891705e0e") : object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:11.355084 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353376 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-var-lib-kubelet\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.355084 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353379 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/199a6bc2-3668-415c-8cc4-813f96eccfa4-agent-certs\") pod \"konnectivity-agent-h8j7v\" (UID: \"199a6bc2-3668-415c-8cc4-813f96eccfa4\") " pod="kube-system/konnectivity-agent-h8j7v" Nov 01 06:09:11.355084 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353408 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr4ql\" (UniqueName: \"kubernetes.io/projected/93f3367a-3295-4f1c-931a-ba9c643d3b76-kube-api-access-xr4ql\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.355084 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353431 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-var-lib-openvswitch\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.355979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353453 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b0855c2-93a3-44a5-9311-63d891705e0e-dbus\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:11.355979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353476 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b856c58-9b9e-47b3-947a-ed56ae892e9b-serviceca\") pod \"node-ca-vth7q\" (UID: \"7b856c58-9b9e-47b3-947a-ed56ae892e9b\") " pod="openshift-image-registry/node-ca-vth7q" Nov 01 06:09:11.355979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353500 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt9qk\" (UniqueName: \"kubernetes.io/projected/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-kube-api-access-qt9qk\") pod \"network-metrics-daemon-b98x2\" (UID: \"8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8\") " pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:11.355979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353526 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-etc-selinux\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.355979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353548 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/199a6bc2-3668-415c-8cc4-813f96eccfa4-konnectivity-ca\") pod \"konnectivity-agent-h8j7v\" (UID: \"199a6bc2-3668-415c-8cc4-813f96eccfa4\") " pod="kube-system/konnectivity-agent-h8j7v" Nov 01 06:09:11.355979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353567 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d48e417b-1b11-4ee3-8b45-0a71f68459bb-cni-binary-copy\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.355979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353573 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/939039b6-e071-4a27-b24e-20bb2669dc15-tmp-dir\") pod \"node-resolver-ks27g\" (UID: \"939039b6-e071-4a27-b24e-20bb2669dc15\") " pod="openshift-dns/node-resolver-ks27g" Nov 01 06:09:11.355979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353567 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b0855c2-93a3-44a5-9311-63d891705e0e-dbus\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:11.355979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353598 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-sysctl-conf\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.355979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353600 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-var-lib-openvswitch\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.355979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353631 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-run-openvswitch\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.355979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353655 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/24ddad96-0a15-4c10-b302-416833e87746-ovnkube-config\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.355979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353679 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-socket-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.355979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353700 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-sys-fs\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.355979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353742 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-cni-dir\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.355979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353762 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-systemd\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.355979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353786 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/24ddad96-0a15-4c10-b302-416833e87746-env-overrides\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.356641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353812 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b623310-9483-47a2-a5de-f19b87819eda-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.356641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353858 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-sys\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.356641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353882 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-var-lib-kubelet\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.356641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353907 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb7kp\" (UniqueName: \"kubernetes.io/projected/b490df30-7bfc-4c25-a9f5-768733b3b6e6-kube-api-access-rb7kp\") pod \"iptables-alerter-rb97g\" (UID: \"b490df30-7bfc-4c25-a9f5-768733b3b6e6\") " pod="openshift-network-operator/iptables-alerter-rb97g" Nov 01 06:09:11.356641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353931 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-run-systemd\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.356641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353958 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-cni-bin\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.356641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353979 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-cni-netd\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.356641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354002 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b856c58-9b9e-47b3-947a-ed56ae892e9b-serviceca\") pod \"node-ca-vth7q\" (UID: \"7b856c58-9b9e-47b3-947a-ed56ae892e9b\") " pod="openshift-image-registry/node-ca-vth7q" Nov 01 06:09:11.356641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354007 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-registration-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.356641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354044 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-sysctl-conf\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.356641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354054 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-socket-dir-parent\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.356641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354060 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-registration-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.356641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354082 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/199a6bc2-3668-415c-8cc4-813f96eccfa4-konnectivity-ca\") pod \"konnectivity-agent-h8j7v\" (UID: \"199a6bc2-3668-415c-8cc4-813f96eccfa4\") " pod="kube-system/konnectivity-agent-h8j7v" Nov 01 06:09:11.356641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354111 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-sys\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.356641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354117 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-socket-dir-parent\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.356641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354191 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-socket-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.356641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354195 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b623310-9483-47a2-a5de-f19b87819eda-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.357158 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354229 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-run-openvswitch\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.357158 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354286 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-var-lib-kubelet\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.357158 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354386 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-etc-selinux\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.357158 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354442 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-run-systemd\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.357158 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.353935 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/939039b6-e071-4a27-b24e-20bb2669dc15-tmp-dir\") pod \"node-resolver-ks27g\" (UID: \"939039b6-e071-4a27-b24e-20bb2669dc15\") " pod="openshift-dns/node-resolver-ks27g" Nov 01 06:09:11.357158 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354459 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-cni-dir\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.357158 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354498 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-sys-fs\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.357158 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354516 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-cni-bin\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.357158 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354528 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b623310-9483-47a2-a5de-f19b87819eda-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.357158 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354554 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-modprobe-d\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.357158 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354557 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-systemd\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.357158 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354556 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-cni-netd\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.357158 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354591 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-kubernetes\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.357158 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354617 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-host\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.357158 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354632 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-modprobe-d\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.357158 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354645 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-run-netns\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.357158 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354669 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-run-multus-certs\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.357158 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354677 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-kubernetes\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.357634 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354693 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z2gw\" (UniqueName: \"kubernetes.io/projected/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-kube-api-access-4z2gw\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.357634 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354718 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd7r9\" (UniqueName: \"kubernetes.io/projected/939039b6-e071-4a27-b24e-20bb2669dc15-kube-api-access-vd7r9\") pod \"node-resolver-ks27g\" (UID: \"939039b6-e071-4a27-b24e-20bb2669dc15\") " pod="openshift-dns/node-resolver-ks27g" Nov 01 06:09:11.357634 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354729 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-run-netns\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.357634 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354744 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-run-ovn\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.357634 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354769 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-log-socket\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.357634 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354793 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-host\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.357634 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354861 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-run-multus-certs\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.357634 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354793 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b623310-9483-47a2-a5de-f19b87819eda-system-cni-dir\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.357634 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354898 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wqjh\" (UniqueName: \"kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh\") pod \"network-check-target-vvpd2\" (UID: \"6228c388-659f-4ca7-8d51-4fa5658424e2\") " pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:11.357634 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354923 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b490df30-7bfc-4c25-a9f5-768733b3b6e6-iptables-alerter-script\") pod \"iptables-alerter-rb97g\" (UID: \"b490df30-7bfc-4c25-a9f5-768733b3b6e6\") " pod="openshift-network-operator/iptables-alerter-rb97g" Nov 01 06:09:11.357634 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354946 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-run-ovn-kubernetes\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.357634 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354971 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vcgd\" (UniqueName: \"kubernetes.io/projected/24ddad96-0a15-4c10-b302-416833e87746-kube-api-access-4vcgd\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.357634 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354976 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/24ddad96-0a15-4c10-b302-416833e87746-env-overrides\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.357634 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.354695 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/24ddad96-0a15-4c10-b302-416833e87746-ovnkube-config\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.357634 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.355037 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-run-ovn\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.357634 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.355042 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-run-ovn-kubernetes\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.357634 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.355096 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b623310-9483-47a2-a5de-f19b87819eda-system-cni-dir\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.358098 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.355248 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-log-socket\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.358098 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.355587 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b623310-9483-47a2-a5de-f19b87819eda-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.358098 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.356377 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b490df30-7bfc-4c25-a9f5-768733b3b6e6-iptables-alerter-script\") pod \"iptables-alerter-rb97g\" (UID: \"b490df30-7bfc-4c25-a9f5-768733b3b6e6\") " pod="openshift-network-operator/iptables-alerter-rb97g" Nov 01 06:09:11.358098 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.356659 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-tuned\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.358098 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.356684 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93f3367a-3295-4f1c-931a-ba9c643d3b76-tmp\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.358098 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.356880 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/199a6bc2-3668-415c-8cc4-813f96eccfa4-agent-certs\") pod \"konnectivity-agent-h8j7v\" (UID: \"199a6bc2-3668-415c-8cc4-813f96eccfa4\") " pod="kube-system/konnectivity-agent-h8j7v" Nov 01 06:09:11.358098 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.356967 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/24ddad96-0a15-4c10-b302-416833e87746-ovn-node-metrics-cert\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.409579 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.409491 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tjfc\" (UniqueName: \"kubernetes.io/projected/d48e417b-1b11-4ee3-8b45-0a71f68459bb-kube-api-access-4tjfc\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.410013 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.409967 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njlqm\" (UniqueName: \"kubernetes.io/projected/7b856c58-9b9e-47b3-947a-ed56ae892e9b-kube-api-access-njlqm\") pod \"node-ca-vth7q\" (UID: \"7b856c58-9b9e-47b3-947a-ed56ae892e9b\") " pod="openshift-image-registry/node-ca-vth7q" Nov 01 06:09:11.410580 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:11.410318 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 06:09:11.410580 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:11.410342 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 06:09:11.410580 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:11.410358 2567 projected.go:196] Error preparing data for projected volume kube-api-access-6wqjh for pod openshift-network-diagnostics/network-check-target-vvpd2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:11.410580 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:11.410436 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh podName:6228c388-659f-4ca7-8d51-4fa5658424e2 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:11.910414286 +0000 UTC m=+2.300599700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6wqjh" (UniqueName: "kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh") pod "network-check-target-vvpd2" (UID: "6228c388-659f-4ca7-8d51-4fa5658424e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:11.411645 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.411613 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt9qk\" (UniqueName: \"kubernetes.io/projected/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-kube-api-access-qt9qk\") pod \"network-metrics-daemon-b98x2\" (UID: \"8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8\") " pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:11.411905 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.411887 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd7r9\" (UniqueName: \"kubernetes.io/projected/939039b6-e071-4a27-b24e-20bb2669dc15-kube-api-access-vd7r9\") pod \"node-resolver-ks27g\" (UID: \"939039b6-e071-4a27-b24e-20bb2669dc15\") " pod="openshift-dns/node-resolver-ks27g" Nov 01 06:09:11.412247 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.412227 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb7kp\" (UniqueName: \"kubernetes.io/projected/b490df30-7bfc-4c25-a9f5-768733b3b6e6-kube-api-access-rb7kp\") pod \"iptables-alerter-rb97g\" (UID: \"b490df30-7bfc-4c25-a9f5-768733b3b6e6\") " pod="openshift-network-operator/iptables-alerter-rb97g" Nov 01 06:09:11.412718 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.412691 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhqdn\" (UniqueName: \"kubernetes.io/projected/3b623310-9483-47a2-a5de-f19b87819eda-kube-api-access-rhqdn\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.412918 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.412899 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr4ql\" (UniqueName: \"kubernetes.io/projected/93f3367a-3295-4f1c-931a-ba9c643d3b76-kube-api-access-xr4ql\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.412918 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.412910 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vcgd\" (UniqueName: \"kubernetes.io/projected/24ddad96-0a15-4c10-b302-416833e87746-kube-api-access-4vcgd\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.414479 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.414454 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z2gw\" (UniqueName: \"kubernetes.io/projected/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-kube-api-access-4z2gw\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.514178 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.514143 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:11.533275 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.533250 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sxts6" Nov 01 06:09:11.547085 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.547055 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h8j7v" Nov 01 06:09:11.564000 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.563974 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ks27g" Nov 01 06:09:11.567757 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.567724 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vth7q" Nov 01 06:09:11.585286 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:11.585245 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b856c58_9b9e_47b3_947a_ed56ae892e9b.slice/crio-a337de93d5a379d6a2691299bc61da059212a2a24a8bdd8604c65e58b4e73735 WatchSource:0}: Error finding container a337de93d5a379d6a2691299bc61da059212a2a24a8bdd8604c65e58b4e73735: Status 404 returned error can't find the container with id a337de93d5a379d6a2691299bc61da059212a2a24a8bdd8604c65e58b4e73735 Nov 01 06:09:11.586266 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:11.586233 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48e417b_1b11_4ee3_8b45_0a71f68459bb.slice/crio-ead6d071a982365a9bbe51da31a7245fab8c10d2afe2cf92acda734f8015b671 WatchSource:0}: Error finding container ead6d071a982365a9bbe51da31a7245fab8c10d2afe2cf92acda734f8015b671: Status 404 returned error can't find the container with id ead6d071a982365a9bbe51da31a7245fab8c10d2afe2cf92acda734f8015b671 Nov 01 06:09:11.587036 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:11.587008 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645c51de28f312aaa75d7c297b03488b.slice/crio-40d91ca4fd54e15d28ffc3f97d8aefa8cbe67c9028c9e3e5c1a7110fc9c84b38 WatchSource:0}: Error finding container 40d91ca4fd54e15d28ffc3f97d8aefa8cbe67c9028c9e3e5c1a7110fc9c84b38: Status 404 returned error can't find the container with id 40d91ca4fd54e15d28ffc3f97d8aefa8cbe67c9028c9e3e5c1a7110fc9c84b38 Nov 01 06:09:11.588214 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:11.588186 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199a6bc2_3668_415c_8cc4_813f96eccfa4.slice/crio-43371c2322f44a7a149eabf907b11ee71d156e9808b02b50c71fdbb4a6f56625 WatchSource:0}: Error finding container 43371c2322f44a7a149eabf907b11ee71d156e9808b02b50c71fdbb4a6f56625: Status 404 returned error can't find the container with id 43371c2322f44a7a149eabf907b11ee71d156e9808b02b50c71fdbb4a6f56625 Nov 01 06:09:11.588995 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:11.588976 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939039b6_e071_4a27_b24e_20bb2669dc15.slice/crio-7a512bd4e2f00eae2e9fb0e81325a8f953c9b5d0f8c7a60f7667351a3fcb3579 WatchSource:0}: Error finding container 7a512bd4e2f00eae2e9fb0e81325a8f953c9b5d0f8c7a60f7667351a3fcb3579: Status 404 returned error can't find the container with id 7a512bd4e2f00eae2e9fb0e81325a8f953c9b5d0f8c7a60f7667351a3fcb3579 Nov 01 06:09:11.596828 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.596806 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:09:11.608982 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:11.608950 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58599ac63e6a6b0b2eb7aede2e6e233a.slice/crio-c0a7ae33424afe6f09493743c78d272fe9a55ac88e1c05518439767aefa7257d WatchSource:0}: Error finding container c0a7ae33424afe6f09493743c78d272fe9a55ac88e1c05518439767aefa7257d: Status 404 returned error can't find the container with id c0a7ae33424afe6f09493743c78d272fe9a55ac88e1c05518439767aefa7257d Nov 01 06:09:11.613174 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.613148 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:11.619001 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.618982 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:11.622262 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:11.622233 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b623310_9483_47a2_a5de_f19b87819eda.slice/crio-a35c2a7d5fb13e8b524daa7d29574a3cebc52e65475679c700dd5b307cfd3de5 WatchSource:0}: Error finding container a35c2a7d5fb13e8b524daa7d29574a3cebc52e65475679c700dd5b307cfd3de5: Status 404 returned error can't find the container with id a35c2a7d5fb13e8b524daa7d29574a3cebc52e65475679c700dd5b307cfd3de5 Nov 01 06:09:11.626807 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:11.626782 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f3367a_3295_4f1c_931a_ba9c643d3b76.slice/crio-de080e1fd8dd4a9ab8aa96a27328db1f10670a87075bc65361d823f2d7b6438f WatchSource:0}: Error finding container de080e1fd8dd4a9ab8aa96a27328db1f10670a87075bc65361d823f2d7b6438f: Status 404 returned error can't find the container with id de080e1fd8dd4a9ab8aa96a27328db1f10670a87075bc65361d823f2d7b6438f Nov 01 06:09:11.632902 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.632887 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rb97g" Nov 01 06:09:11.642626 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:11.642600 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb490df30_7bfc_4c25_a9f5_768733b3b6e6.slice/crio-b213f89ca630f32c262ff8dca5cd1c7e79b4648ae2b632270c942b8e74d90ce3 WatchSource:0}: Error finding container b213f89ca630f32c262ff8dca5cd1c7e79b4648ae2b632270c942b8e74d90ce3: Status 404 returned error can't find the container with id b213f89ca630f32c262ff8dca5cd1c7e79b4648ae2b632270c942b8e74d90ce3 Nov 01 06:09:11.652602 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.652579 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:11.662573 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:11.662545 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ddad96_0a15_4c10_b302_416833e87746.slice/crio-380de1f4b09bb1aa0a4e1a4e6fc44c27dc61d8eb48f5afe09509e396c0859d5f WatchSource:0}: Error finding container 380de1f4b09bb1aa0a4e1a4e6fc44c27dc61d8eb48f5afe09509e396c0859d5f: Status 404 returned error can't find the container with id 380de1f4b09bb1aa0a4e1a4e6fc44c27dc61d8eb48f5afe09509e396c0859d5f Nov 01 06:09:11.858552 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.858504 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs\") pod \"network-metrics-daemon-b98x2\" (UID: \"8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8\") " pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:11.858552 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.858557 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:11.858820 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:11.858664 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:11.858820 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:11.858704 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:11.858820 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:11.858716 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret podName:5b0855c2-93a3-44a5-9311-63d891705e0e nodeName:}" failed. No retries permitted until 2026-11-01 06:09:12.85870146 +0000 UTC m=+3.248886868 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret") pod "global-pull-secret-syncer-txvqp" (UID: "5b0855c2-93a3-44a5-9311-63d891705e0e") : object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:11.858820 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:11.858792 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs podName:8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:12.858771834 +0000 UTC m=+3.248957229 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs") pod "network-metrics-daemon-b98x2" (UID: "8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:11.959064 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:11.958963 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wqjh\" (UniqueName: \"kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh\") pod \"network-check-target-vvpd2\" (UID: \"6228c388-659f-4ca7-8d51-4fa5658424e2\") " pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:11.959783 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:11.959151 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 06:09:11.959783 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:11.959167 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 06:09:11.959783 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:11.959177 2567 projected.go:196] Error preparing data for projected volume kube-api-access-6wqjh for pod openshift-network-diagnostics/network-check-target-vvpd2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:11.959783 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:11.959233 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh podName:6228c388-659f-4ca7-8d51-4fa5658424e2 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:12.959218875 +0000 UTC m=+3.349404268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6wqjh" (UniqueName: "kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh") pod "network-check-target-vvpd2" (UID: "6228c388-659f-4ca7-8d51-4fa5658424e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:12.086637 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:12.086577 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ks27g" event={"ID":"939039b6-e071-4a27-b24e-20bb2669dc15","Type":"ContainerStarted","Data":"7a512bd4e2f00eae2e9fb0e81325a8f953c9b5d0f8c7a60f7667351a3fcb3579"} Nov 01 06:09:12.088354 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:12.088328 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal" event={"ID":"645c51de28f312aaa75d7c297b03488b","Type":"ContainerStarted","Data":"40d91ca4fd54e15d28ffc3f97d8aefa8cbe67c9028c9e3e5c1a7110fc9c84b38"} Nov 01 06:09:12.089498 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:12.089474 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h8j7v" event={"ID":"199a6bc2-3668-415c-8cc4-813f96eccfa4","Type":"ContainerStarted","Data":"43371c2322f44a7a149eabf907b11ee71d156e9808b02b50c71fdbb4a6f56625"} Nov 01 06:09:12.090607 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:12.090588 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sxts6" event={"ID":"d48e417b-1b11-4ee3-8b45-0a71f68459bb","Type":"ContainerStarted","Data":"ead6d071a982365a9bbe51da31a7245fab8c10d2afe2cf92acda734f8015b671"} Nov 01 06:09:12.091835 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:12.091813 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" event={"ID":"24ddad96-0a15-4c10-b302-416833e87746","Type":"ContainerStarted","Data":"380de1f4b09bb1aa0a4e1a4e6fc44c27dc61d8eb48f5afe09509e396c0859d5f"} Nov 01 06:09:12.092984 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:12.092963 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rb97g" event={"ID":"b490df30-7bfc-4c25-a9f5-768733b3b6e6","Type":"ContainerStarted","Data":"b213f89ca630f32c262ff8dca5cd1c7e79b4648ae2b632270c942b8e74d90ce3"} Nov 01 06:09:12.094025 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:12.094004 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-258pf" event={"ID":"93f3367a-3295-4f1c-931a-ba9c643d3b76","Type":"ContainerStarted","Data":"de080e1fd8dd4a9ab8aa96a27328db1f10670a87075bc65361d823f2d7b6438f"} Nov 01 06:09:12.095237 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:12.095215 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fvnw6" event={"ID":"3b623310-9483-47a2-a5de-f19b87819eda","Type":"ContainerStarted","Data":"a35c2a7d5fb13e8b524daa7d29574a3cebc52e65475679c700dd5b307cfd3de5"} Nov 01 06:09:12.096252 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:12.096234 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" event={"ID":"58599ac63e6a6b0b2eb7aede2e6e233a","Type":"ContainerStarted","Data":"c0a7ae33424afe6f09493743c78d272fe9a55ac88e1c05518439767aefa7257d"} Nov 01 06:09:12.097304 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:12.097284 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vth7q" event={"ID":"7b856c58-9b9e-47b3-947a-ed56ae892e9b","Type":"ContainerStarted","Data":"a337de93d5a379d6a2691299bc61da059212a2a24a8bdd8604c65e58b4e73735"} Nov 01 06:09:12.111401 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:12.111368 2567 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 06:09:12.321086 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:12.320992 2567 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 06:09:12.339532 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:12.339464 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 06:04:11 +0000 UTC" deadline="2028-04-17 01:46:54.044210738 +0000 UTC" Nov 01 06:09:12.339532 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:12.339526 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12787h37m41.704687467s" Nov 01 06:09:12.402750 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:12.402708 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e17cd4c_13b9_4dbf_8798_5fc68eaa1271.slice/crio-a5599496786fe856313aeae337a6feb17eae0eaf383dedf77fda113fae056a87 WatchSource:0}: Error finding container a5599496786fe856313aeae337a6feb17eae0eaf383dedf77fda113fae056a87: Status 404 returned error can't find the container with id a5599496786fe856313aeae337a6feb17eae0eaf383dedf77fda113fae056a87 Nov 01 06:09:12.509219 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:12.509180 2567 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 06:09:12.866343 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:12.866301 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs\") pod \"network-metrics-daemon-b98x2\" (UID: \"8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8\") " pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:12.866546 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:12.866374 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:12.866614 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:12.866556 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:12.866676 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:12.866624 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret podName:5b0855c2-93a3-44a5-9311-63d891705e0e nodeName:}" failed. No retries permitted until 2026-11-01 06:09:14.866604698 +0000 UTC m=+5.256790112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret") pod "global-pull-secret-syncer-txvqp" (UID: "5b0855c2-93a3-44a5-9311-63d891705e0e") : object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:12.866741 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:12.866691 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:12.866741 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:12.866724 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs podName:8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:14.866712427 +0000 UTC m=+5.256897837 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs") pod "network-metrics-daemon-b98x2" (UID: "8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:12.967859 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:12.967816 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wqjh\" (UniqueName: \"kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh\") pod \"network-check-target-vvpd2\" (UID: \"6228c388-659f-4ca7-8d51-4fa5658424e2\") " pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:12.968424 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:12.968009 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 06:09:12.968424 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:12.968032 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 06:09:12.968424 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:12.968044 2567 projected.go:196] Error preparing data for projected volume kube-api-access-6wqjh for pod openshift-network-diagnostics/network-check-target-vvpd2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:12.968424 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:12.968111 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh podName:6228c388-659f-4ca7-8d51-4fa5658424e2 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:14.968092148 +0000 UTC m=+5.358277550 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6wqjh" (UniqueName: "kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh") pod "network-check-target-vvpd2" (UID: "6228c388-659f-4ca7-8d51-4fa5658424e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:13.082745 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:13.082205 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:13.082745 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:13.082364 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b98x2" podUID="8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8" Nov 01 06:09:13.082745 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:13.082457 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:13.082745 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:13.082536 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvpd2" podUID="6228c388-659f-4ca7-8d51-4fa5658424e2" Nov 01 06:09:13.082745 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:13.082601 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:13.082745 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:13.082675 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txvqp" podUID="5b0855c2-93a3-44a5-9311-63d891705e0e" Nov 01 06:09:13.113211 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:13.113167 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" event={"ID":"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271","Type":"ContainerStarted","Data":"a5599496786fe856313aeae337a6feb17eae0eaf383dedf77fda113fae056a87"} Nov 01 06:09:13.340774 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:13.340637 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 06:04:11 +0000 UTC" deadline="2028-04-20 20:15:49.707950685 +0000 UTC" Nov 01 06:09:13.340774 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:13.340679 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12878h6m36.367276098s" Nov 01 06:09:14.885346 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:14.885300 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs\") pod \"network-metrics-daemon-b98x2\" (UID: \"8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8\") " pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:14.885833 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:14.885374 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:14.885833 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:14.885542 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:14.885833 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:14.885613 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret podName:5b0855c2-93a3-44a5-9311-63d891705e0e nodeName:}" failed. No retries permitted until 2026-11-01 06:09:18.885592162 +0000 UTC m=+9.275777579 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret") pod "global-pull-secret-syncer-txvqp" (UID: "5b0855c2-93a3-44a5-9311-63d891705e0e") : object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:14.885833 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:14.885680 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:14.885833 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:14.885717 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs podName:8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:18.885703704 +0000 UTC m=+9.275889113 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs") pod "network-metrics-daemon-b98x2" (UID: "8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:14.986514 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:14.986472 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wqjh\" (UniqueName: \"kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh\") pod \"network-check-target-vvpd2\" (UID: \"6228c388-659f-4ca7-8d51-4fa5658424e2\") " pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:14.986737 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:14.986702 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 06:09:14.986737 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:14.986736 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 06:09:14.986879 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:14.986751 2567 projected.go:196] Error preparing data for projected volume kube-api-access-6wqjh for pod openshift-network-diagnostics/network-check-target-vvpd2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:14.986879 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:14.986819 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh podName:6228c388-659f-4ca7-8d51-4fa5658424e2 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:18.986798723 +0000 UTC m=+9.376984139 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6wqjh" (UniqueName: "kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh") pod "network-check-target-vvpd2" (UID: "6228c388-659f-4ca7-8d51-4fa5658424e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:15.084820 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:15.084231 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:15.084820 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:15.084364 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txvqp" podUID="5b0855c2-93a3-44a5-9311-63d891705e0e" Nov 01 06:09:15.084820 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:15.084478 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:15.084820 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:15.084562 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b98x2" podUID="8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8" Nov 01 06:09:15.084820 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:15.084628 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:15.084820 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:15.084741 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvpd2" podUID="6228c388-659f-4ca7-8d51-4fa5658424e2" Nov 01 06:09:16.137911 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:16.137621 2567 generic.go:356] "Generic (PLEG): container finished" podID="3b623310-9483-47a2-a5de-f19b87819eda" containerID="275a4717cc5ae5dfc425b7c32987dfbb88c1b0fdc174ae4817005fe76ebbe4d5" exitCode=0 Nov 01 06:09:16.138413 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:16.137948 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fvnw6" event={"ID":"3b623310-9483-47a2-a5de-f19b87819eda","Type":"ContainerDied","Data":"275a4717cc5ae5dfc425b7c32987dfbb88c1b0fdc174ae4817005fe76ebbe4d5"} Nov 01 06:09:17.081665 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:17.081632 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:17.081820 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:17.081765 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvpd2" podUID="6228c388-659f-4ca7-8d51-4fa5658424e2" Nov 01 06:09:17.081905 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:17.081824 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:17.081974 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:17.081924 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txvqp" podUID="5b0855c2-93a3-44a5-9311-63d891705e0e" Nov 01 06:09:17.082045 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:17.081972 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:17.082093 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:17.082057 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b98x2" podUID="8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8" Nov 01 06:09:17.150051 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:17.150006 2567 generic.go:356] "Generic (PLEG): container finished" podID="58599ac63e6a6b0b2eb7aede2e6e233a" containerID="c7a26ddd76fe46720d9c0356a72dc73bc750c01e016da41f2fff42257c167da6" exitCode=0 Nov 01 06:09:17.150504 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:17.150422 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" event={"ID":"58599ac63e6a6b0b2eb7aede2e6e233a","Type":"ContainerDied","Data":"c7a26ddd76fe46720d9c0356a72dc73bc750c01e016da41f2fff42257c167da6"} Nov 01 06:09:17.156518 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:17.156487 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal" event={"ID":"645c51de28f312aaa75d7c297b03488b","Type":"ContainerStarted","Data":"b52f53e5e2278966472f585f97b01eaed9dfa2a6d3c3081ddada2d96b55827fe"} Nov 01 06:09:17.175539 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:17.175502 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h8j7v" event={"ID":"199a6bc2-3668-415c-8cc4-813f96eccfa4","Type":"ContainerStarted","Data":"00c29f867d9c242cfb64ce54c7c0c1a9a83bf3710d5773b3de2f64643c461073"} Nov 01 06:09:17.376157 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:17.376091 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h8j7v" podStartSLOduration=2.028135594 podStartE2EDuration="7.37607135s" totalImagesPullingTime="5.347935756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:17.374563645 +0000 UTC m=+7.764749057" watchObservedRunningTime="2026-11-01 06:09:17.37607135 +0000 UTC m=+7.766256766" Nov 01 06:09:18.183652 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:18.183611 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" event={"ID":"58599ac63e6a6b0b2eb7aede2e6e233a","Type":"ContainerStarted","Data":"a028da7f9fbd18a3456c0944599c78c30ac81f562993e184222f9cbbaf2c6129"} Nov 01 06:09:18.187785 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:18.187745 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vth7q" event={"ID":"7b856c58-9b9e-47b3-947a-ed56ae892e9b","Type":"ContainerStarted","Data":"be05d3865c643ced9124a86b2dcab52a40df00c120e3caeab222c8b97bd9992b"} Nov 01 06:09:18.295945 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:18.295822 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal" podStartSLOduration=8.295759613 podStartE2EDuration="8.295759613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:17.490385338 +0000 UTC m=+7.880570774" watchObservedRunningTime="2026-11-01 06:09:18.295759613 +0000 UTC m=+8.685945027" Nov 01 06:09:18.296117 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:18.295947 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" podStartSLOduration=8.295938049 podStartE2EDuration="8.295938049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:18.295627415 +0000 UTC m=+8.685812831" watchObservedRunningTime="2026-11-01 06:09:18.295938049 +0000 UTC m=+8.686123465" Nov 01 06:09:18.923382 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:18.923341 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:18.923585 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:18.923443 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs\") pod \"network-metrics-daemon-b98x2\" (UID: \"8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8\") " pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:18.923585 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:18.923565 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:18.923710 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:18.923612 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs podName:8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:26.923600033 +0000 UTC m=+17.313785438 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs") pod "network-metrics-daemon-b98x2" (UID: "8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:18.923710 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:18.923671 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:18.923710 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:18.923693 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret podName:5b0855c2-93a3-44a5-9311-63d891705e0e nodeName:}" failed. No retries permitted until 2026-11-01 06:09:26.923686787 +0000 UTC m=+17.313872179 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret") pod "global-pull-secret-syncer-txvqp" (UID: "5b0855c2-93a3-44a5-9311-63d891705e0e") : object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:19.024066 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:19.024018 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wqjh\" (UniqueName: \"kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh\") pod \"network-check-target-vvpd2\" (UID: \"6228c388-659f-4ca7-8d51-4fa5658424e2\") " pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:19.024251 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:19.024209 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 06:09:19.024251 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:19.024229 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 06:09:19.024251 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:19.024249 2567 projected.go:196] Error preparing data for projected volume kube-api-access-6wqjh for pod openshift-network-diagnostics/network-check-target-vvpd2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:19.024417 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:19.024301 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh podName:6228c388-659f-4ca7-8d51-4fa5658424e2 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:27.02428512 +0000 UTC m=+17.414470513 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6wqjh" (UniqueName: "kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh") pod "network-check-target-vvpd2" (UID: "6228c388-659f-4ca7-8d51-4fa5658424e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:19.082053 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:19.081974 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:19.082228 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:19.082119 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvpd2" podUID="6228c388-659f-4ca7-8d51-4fa5658424e2" Nov 01 06:09:19.082228 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:19.082194 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:19.082337 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:19.082274 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txvqp" podUID="5b0855c2-93a3-44a5-9311-63d891705e0e" Nov 01 06:09:19.082399 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:19.082335 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:19.082451 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:19.082419 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b98x2" podUID="8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8" Nov 01 06:09:20.201818 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:20.201773 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" event={"ID":"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271","Type":"ContainerStarted","Data":"f6b158a8f1e4870bb76941d24c5d4e2c8770e7ac387abdad0907fd8bc4c97bca"} Nov 01 06:09:20.935122 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:20.934273 2567 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h8j7v" Nov 01 06:09:20.935122 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:20.934907 2567 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h8j7v" Nov 01 06:09:21.047558 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:21.047497 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vth7q" podStartSLOduration=4.701920497 podStartE2EDuration="11.047476919s" totalImagesPullingTime="6.345556422s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:18.40753591 +0000 UTC m=+8.797721326" watchObservedRunningTime="2026-11-01 06:09:21.047476919 +0000 UTC m=+11.437662335" Nov 01 06:09:21.082185 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:21.082146 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:21.082367 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:21.082268 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvpd2" podUID="6228c388-659f-4ca7-8d51-4fa5658424e2" Nov 01 06:09:21.082367 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:21.082330 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:21.082480 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:21.082386 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b98x2" podUID="8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8" Nov 01 06:09:21.082480 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:21.082426 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:21.082480 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:21.082472 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txvqp" podUID="5b0855c2-93a3-44a5-9311-63d891705e0e" Nov 01 06:09:21.207437 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:21.207210 2567 generic.go:356] "Generic (PLEG): container finished" podID="3b623310-9483-47a2-a5de-f19b87819eda" containerID="e900319af2df00f1628e273931cd34643fb472938c77bb8bd34e830259dfd95e" exitCode=0 Nov 01 06:09:21.207437 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:21.207309 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fvnw6" event={"ID":"3b623310-9483-47a2-a5de-f19b87819eda","Type":"ContainerDied","Data":"e900319af2df00f1628e273931cd34643fb472938c77bb8bd34e830259dfd95e"} Nov 01 06:09:21.209548 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:21.209184 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ks27g" event={"ID":"939039b6-e071-4a27-b24e-20bb2669dc15","Type":"ContainerStarted","Data":"71e2e8a60241276795811d3a14149e28c1ac387790e2a7c1f6e53e1476f8980b"} Nov 01 06:09:21.211987 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:21.211963 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sxts6" event={"ID":"d48e417b-1b11-4ee3-8b45-0a71f68459bb","Type":"ContainerStarted","Data":"9f1e08e9170b20ef396716bf2fa2140be71d481d5edaa38bd68fc3bf83a24b02"} Nov 01 06:09:21.350994 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:21.350968 2567 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 06:09:21.436685 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:21.436629 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-ks27g" podStartSLOduration=2.32729814 podStartE2EDuration="11.436611521s" totalImagesPullingTime="9.109313381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:21.436502418 +0000 UTC m=+11.826687833" watchObservedRunningTime="2026-11-01 06:09:21.436611521 +0000 UTC m=+11.826796937" Nov 01 06:09:21.551159 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:21.551050 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-sxts6" podStartSLOduration=2.454956136 podStartE2EDuration="11.551033782s" totalImagesPullingTime="9.096077646s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:21.550447172 +0000 UTC m=+11.940632586" watchObservedRunningTime="2026-11-01 06:09:21.551033782 +0000 UTC m=+11.941219204" Nov 01 06:09:22.016911 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:22.016779 2567 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T06:09:21.350987326Z","UUID":"aea1d73e-ffb0-4a6f-b8f3-bc653e830546","Handler":null,"Name":"","Endpoint":""} Nov 01 06:09:22.019206 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:22.019184 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 06:09:22.019364 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:22.019217 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 06:09:22.216107 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:22.215958 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" event={"ID":"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271","Type":"ContainerStarted","Data":"8ab924bbeffb0e6c09c3f172737accab757f197fd43c52e48b645fbfdb62e1cd"} Nov 01 06:09:23.082348 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:23.082119 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:23.082510 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:23.082168 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:23.082510 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:23.082426 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b98x2" podUID="8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8" Nov 01 06:09:23.082510 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:23.082176 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:23.082510 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:23.082455 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txvqp" podUID="5b0855c2-93a3-44a5-9311-63d891705e0e" Nov 01 06:09:23.082649 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:23.082512 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvpd2" podUID="6228c388-659f-4ca7-8d51-4fa5658424e2" Nov 01 06:09:23.219754 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:23.219713 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rb97g" event={"ID":"b490df30-7bfc-4c25-a9f5-768733b3b6e6","Type":"ContainerStarted","Data":"2d242c4e1a8d014ea3fe21596ed92e41db4f13707fb9ee0cb611ae29a4c54af9"} Nov 01 06:09:23.332886 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:23.332774 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rb97g" podStartSLOduration=3.755229997 podStartE2EDuration="13.332760751s" totalImagesPullingTime="9.577530754s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:23.332537719 +0000 UTC m=+13.722723132" watchObservedRunningTime="2026-11-01 06:09:23.332760751 +0000 UTC m=+13.722946165" Nov 01 06:09:24.223524 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:24.223485 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-258pf" event={"ID":"93f3367a-3295-4f1c-931a-ba9c643d3b76","Type":"ContainerStarted","Data":"57522213fe91ba7f6d41e8f3ad33362024a3f848ce5699f0c16ef3bbb5778b0c"} Nov 01 06:09:24.225899 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:24.225870 2567 generic.go:356] "Generic (PLEG): container finished" podID="3b623310-9483-47a2-a5de-f19b87819eda" containerID="dd7533cbebfea44f33a150517560396b834415ed3d606dc0e2172becd127bcca" exitCode=0 Nov 01 06:09:24.226044 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:24.225967 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fvnw6" event={"ID":"3b623310-9483-47a2-a5de-f19b87819eda","Type":"ContainerDied","Data":"dd7533cbebfea44f33a150517560396b834415ed3d606dc0e2172becd127bcca"} Nov 01 06:09:24.228534 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:24.228513 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" event={"ID":"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271","Type":"ContainerStarted","Data":"1c73875ab35a908150e732fe85b77b7a0b318438f6b48ddad063a6eac1204ce7"} Nov 01 06:09:24.336151 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:24.336098 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-258pf" podStartSLOduration=2.505636531 podStartE2EDuration="14.336077837s" totalImagesPullingTime="11.830441306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:24.335707424 +0000 UTC m=+14.725892837" watchObservedRunningTime="2026-11-01 06:09:24.336077837 +0000 UTC m=+14.726263250" Nov 01 06:09:24.450423 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:24.450359 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" podStartSLOduration=3.745437383 podStartE2EDuration="14.450340654s" totalImagesPullingTime="10.704903271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:24.450318466 +0000 UTC m=+14.840503881" watchObservedRunningTime="2026-11-01 06:09:24.450340654 +0000 UTC m=+14.840526062" Nov 01 06:09:25.081985 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:25.081945 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:25.082155 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:25.081999 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:25.082155 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:25.082102 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvpd2" podUID="6228c388-659f-4ca7-8d51-4fa5658424e2" Nov 01 06:09:25.082155 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:25.082106 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:25.082275 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:25.082204 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b98x2" podUID="8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8" Nov 01 06:09:25.082314 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:25.082288 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txvqp" podUID="5b0855c2-93a3-44a5-9311-63d891705e0e" Nov 01 06:09:26.236896 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:26.236859 2567 generic.go:356] "Generic (PLEG): container finished" podID="3b623310-9483-47a2-a5de-f19b87819eda" containerID="7f03e1809f418a0a1da6fdf456a8ad0a45a4207b287fa37a41e858932aa15e4c" exitCode=0 Nov 01 06:09:26.237664 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:26.236932 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fvnw6" event={"ID":"3b623310-9483-47a2-a5de-f19b87819eda","Type":"ContainerDied","Data":"7f03e1809f418a0a1da6fdf456a8ad0a45a4207b287fa37a41e858932aa15e4c"} Nov 01 06:09:26.992432 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:26.992178 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs\") pod \"network-metrics-daemon-b98x2\" (UID: \"8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8\") " pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:26.992621 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:26.992467 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:26.992621 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:26.992335 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:26.992621 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:26.992566 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs podName:8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:42.992547473 +0000 UTC m=+33.382732882 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs") pod "network-metrics-daemon-b98x2" (UID: "8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:26.992621 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:26.992620 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:26.992761 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:26.992671 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret podName:5b0855c2-93a3-44a5-9311-63d891705e0e nodeName:}" failed. No retries permitted until 2026-11-01 06:09:42.992655642 +0000 UTC m=+33.382841051 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret") pod "global-pull-secret-syncer-txvqp" (UID: "5b0855c2-93a3-44a5-9311-63d891705e0e") : object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:27.081864 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:27.081812 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:27.081864 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:27.081861 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:27.082093 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:27.081976 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvpd2" podUID="6228c388-659f-4ca7-8d51-4fa5658424e2" Nov 01 06:09:27.082093 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:27.082056 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txvqp" podUID="5b0855c2-93a3-44a5-9311-63d891705e0e" Nov 01 06:09:27.082093 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:27.082075 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:27.082203 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:27.082154 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b98x2" podUID="8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8" Nov 01 06:09:27.093080 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:27.093045 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wqjh\" (UniqueName: \"kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh\") pod \"network-check-target-vvpd2\" (UID: \"6228c388-659f-4ca7-8d51-4fa5658424e2\") " pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:27.093242 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:27.093224 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 06:09:27.093302 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:27.093252 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 06:09:27.093302 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:27.093268 2567 projected.go:196] Error preparing data for projected volume kube-api-access-6wqjh for pod openshift-network-diagnostics/network-check-target-vvpd2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:27.093385 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:27.093350 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh podName:6228c388-659f-4ca7-8d51-4fa5658424e2 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:43.093327135 +0000 UTC m=+33.483512547 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6wqjh" (UniqueName: "kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh") pod "network-check-target-vvpd2" (UID: "6228c388-659f-4ca7-8d51-4fa5658424e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:28.244702 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:28.244464 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" event={"ID":"24ddad96-0a15-4c10-b302-416833e87746","Type":"ContainerStarted","Data":"b5c50d10a170f957ca114e41cd131e10e8554534088ee771cb1021698413ffe9"} Nov 01 06:09:28.245141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:28.244763 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" event={"ID":"24ddad96-0a15-4c10-b302-416833e87746","Type":"ContainerStarted","Data":"c27a250663f147bcc37bcd4ceeff003e08db52ee3b41a0a6fdbd8fd2f9ee27b6"} Nov 01 06:09:28.245141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:28.244800 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" event={"ID":"24ddad96-0a15-4c10-b302-416833e87746","Type":"ContainerStarted","Data":"8120f6acfb13ba8465bc3aa1588250dea6f729f8c0c4dc932ce26e8b1070b5de"} Nov 01 06:09:28.245141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:28.244865 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" event={"ID":"24ddad96-0a15-4c10-b302-416833e87746","Type":"ContainerStarted","Data":"32c124e48cfed05237cbbc822c46a54df6e657e9c871f6bbafcb0b5a2dd76823"} Nov 01 06:09:28.245141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:28.244896 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" event={"ID":"24ddad96-0a15-4c10-b302-416833e87746","Type":"ContainerStarted","Data":"8ad830ce794a2a8ef32f688b9cdb776219ed4fc22367b4bd391d5d1225210c10"} Nov 01 06:09:28.245141 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:28.244940 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" event={"ID":"24ddad96-0a15-4c10-b302-416833e87746","Type":"ContainerStarted","Data":"b37122698aa306e071912395b921d1036563bbf24a364ddfbac831fd0dc16b65"} Nov 01 06:09:28.247237 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:28.247212 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fvnw6" event={"ID":"3b623310-9483-47a2-a5de-f19b87819eda","Type":"ContainerStarted","Data":"90e150b3ba937632ac7584524dbac6844faf1fd36a383ee2c397e66a01a4b403"} Nov 01 06:09:29.081951 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:29.081913 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:29.081951 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:29.081943 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:29.082172 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:29.082028 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txvqp" podUID="5b0855c2-93a3-44a5-9311-63d891705e0e" Nov 01 06:09:29.082172 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:29.082073 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:29.082172 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:29.082147 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvpd2" podUID="6228c388-659f-4ca7-8d51-4fa5658424e2" Nov 01 06:09:29.082300 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:29.082237 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b98x2" podUID="8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8" Nov 01 06:09:29.251973 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:29.251934 2567 generic.go:356] "Generic (PLEG): container finished" podID="3b623310-9483-47a2-a5de-f19b87819eda" containerID="90e150b3ba937632ac7584524dbac6844faf1fd36a383ee2c397e66a01a4b403" exitCode=0 Nov 01 06:09:29.252439 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:29.251997 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fvnw6" event={"ID":"3b623310-9483-47a2-a5de-f19b87819eda","Type":"ContainerDied","Data":"90e150b3ba937632ac7584524dbac6844faf1fd36a383ee2c397e66a01a4b403"} Nov 01 06:09:30.258048 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:30.258003 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" event={"ID":"24ddad96-0a15-4c10-b302-416833e87746","Type":"ContainerStarted","Data":"c258fc52f88f4026abc1222c2b6caa7813e2e269054e16cd372824c6143eb33c"} Nov 01 06:09:30.260825 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:30.260799 2567 generic.go:356] "Generic (PLEG): container finished" podID="3b623310-9483-47a2-a5de-f19b87819eda" containerID="4e633c9b8d58ddfe900be37963120338e74861b5c539c76b15b21c45c67ed06d" exitCode=0 Nov 01 06:09:30.260936 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:30.260864 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fvnw6" event={"ID":"3b623310-9483-47a2-a5de-f19b87819eda","Type":"ContainerDied","Data":"4e633c9b8d58ddfe900be37963120338e74861b5c539c76b15b21c45c67ed06d"} Nov 01 06:09:30.281848 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:30.281804 2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h8j7v" Nov 01 06:09:30.282415 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:30.281970 2567 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 06:09:30.282518 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:30.282491 2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h8j7v" Nov 01 06:09:31.081641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:31.081611 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:31.081641 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:31.081629 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:31.081828 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:31.081654 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:31.081828 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:31.081732 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txvqp" podUID="5b0855c2-93a3-44a5-9311-63d891705e0e" Nov 01 06:09:31.081938 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:31.081876 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b98x2" podUID="8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8" Nov 01 06:09:31.081982 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:31.081949 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvpd2" podUID="6228c388-659f-4ca7-8d51-4fa5658424e2" Nov 01 06:09:31.266275 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:31.266236 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fvnw6" event={"ID":"3b623310-9483-47a2-a5de-f19b87819eda","Type":"ContainerStarted","Data":"17f71a973cb05978bd4576d8844c8b2182457787f21dba34157c148ad5d39aea"} Nov 01 06:09:32.272597 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:32.272561 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" event={"ID":"24ddad96-0a15-4c10-b302-416833e87746","Type":"ContainerStarted","Data":"a4e0228f4464e5684dfafe141f77f1d51eaa61f4a7e6c0cf19f406d9e19eabbc"} Nov 01 06:09:32.543934 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:32.543814 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fvnw6" podStartSLOduration=9.38429368 podStartE2EDuration="22.543801589s" totalImagesPullingTime="13.159507909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:31.431163822 +0000 UTC m=+21.821349278" watchObservedRunningTime="2026-11-01 06:09:32.543801589 +0000 UTC m=+22.933987003" Nov 01 06:09:32.544218 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:32.544058 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" podStartSLOduration=6.523186218 podStartE2EDuration="22.544052514s" totalImagesPullingTime="16.020866296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:32.543655375 +0000 UTC m=+22.933840803" watchObservedRunningTime="2026-11-01 06:09:32.544052514 +0000 UTC m=+22.934237928" Nov 01 06:09:33.082646 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:33.082606 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:33.082805 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:33.082672 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:33.082805 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:33.082766 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:33.082805 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:33.082782 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txvqp" podUID="5b0855c2-93a3-44a5-9311-63d891705e0e" Nov 01 06:09:33.082957 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:33.082872 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b98x2" podUID="8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8" Nov 01 06:09:33.082957 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:33.082944 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvpd2" podUID="6228c388-659f-4ca7-8d51-4fa5658424e2" Nov 01 06:09:33.274786 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:33.274758 2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:33.275225 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:33.274853 2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:33.275225 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:33.274907 2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:33.291478 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:33.291451 2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:33.291628 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:33.291558 2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:34.994425 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:34.994390 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-txvqp"] Nov 01 06:09:34.994980 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:34.994954 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vvpd2"] Nov 01 06:09:34.995091 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:34.995062 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:34.995150 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:34.995084 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:34.995208 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:34.995181 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvpd2" podUID="6228c388-659f-4ca7-8d51-4fa5658424e2" Nov 01 06:09:34.995303 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:34.995284 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txvqp" podUID="5b0855c2-93a3-44a5-9311-63d891705e0e" Nov 01 06:09:35.043727 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:35.043689 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b98x2"] Nov 01 06:09:35.043929 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:35.043819 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:35.043996 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:35.043948 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b98x2" podUID="8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8" Nov 01 06:09:35.212808 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:35.212772 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:35.402583 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:35.402547 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:36.525872 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:36.525766 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:36.675148 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:36.673973 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:37.081849 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:37.081814 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:37.082032 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:37.081929 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvpd2" podUID="6228c388-659f-4ca7-8d51-4fa5658424e2" Nov 01 06:09:37.082032 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:37.082012 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:37.082124 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:37.082107 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txvqp" podUID="5b0855c2-93a3-44a5-9311-63d891705e0e" Nov 01 06:09:37.082166 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:37.082152 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:37.082245 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:37.082231 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b98x2" podUID="8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8" Nov 01 06:09:39.082188 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:39.082147 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:39.082188 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:39.082168 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:39.082666 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:39.082155 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:39.082666 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:39.082261 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvpd2" podUID="6228c388-659f-4ca7-8d51-4fa5658424e2" Nov 01 06:09:39.082666 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:39.082343 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txvqp" podUID="5b0855c2-93a3-44a5-9311-63d891705e0e" Nov 01 06:09:39.082666 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:39.082409 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b98x2" podUID="8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8" Nov 01 06:09:39.164017 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:39.163988 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:39.325580 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:39.325534 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:40.088080 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:40.088043 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:40.380171 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:40.380137 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:40.538179 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:40.538150 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:40.680637 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:40.680560 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:40.829997 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:40.829965 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeReady" Nov 01 06:09:40.830169 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:40.830109 2567 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 06:09:40.833489 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:40.833467 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:40.981940 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:40.981823 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-62hgp"] Nov 01 06:09:40.984857 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:40.984804 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:40.988174 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:40.988153 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2vfjz"] Nov 01 06:09:40.988278 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:40.988266 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:40.992529 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:40.992470 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b9rlh"] Nov 01 06:09:40.992674 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:40.992611 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2vfjz" Nov 01 06:09:40.994977 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:40.994958 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:41.007619 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.007590 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-62hgp"] Nov 01 06:09:41.016959 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.016914 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2vfjz"] Nov 01 06:09:41.017501 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.017478 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b9rlh"] Nov 01 06:09:41.040353 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.040327 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 06:09:41.040919 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.040895 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 06:09:41.041088 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.041067 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jgqp9\"" Nov 01 06:09:41.044341 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.044315 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 06:09:41.044484 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.044420 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8pggw\"" Nov 01 06:09:41.044484 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.044436 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 06:09:41.044589 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.044574 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 06:09:41.046522 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.046506 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4w6wr\"" Nov 01 06:09:41.056159 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.056142 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 06:09:41.056159 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.056144 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 06:09:41.056286 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.056168 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 06:09:41.056286 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.056219 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 06:09:41.082541 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.082514 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:41.082631 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.082591 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:41.082697 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.082684 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:41.091897 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.091872 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xd5q\" (UniqueName: \"kubernetes.io/projected/45331df5-857e-4663-9131-6575bcd3095a-kube-api-access-5xd5q\") pod \"ingress-canary-2vfjz\" (UID: \"45331df5-857e-4663-9131-6575bcd3095a\") " pod="openshift-ingress-canary/ingress-canary-2vfjz" Nov 01 06:09:41.092201 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.091907 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/568704c3-e334-44e1-8373-12a873c305b5-config-volume\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:41.092201 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.091937 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:41.092201 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.091989 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5pnm\" (UniqueName: \"kubernetes.io/projected/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-api-access-n5pnm\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:41.092201 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.092080 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45331df5-857e-4663-9131-6575bcd3095a-cert\") pod \"ingress-canary-2vfjz\" (UID: \"45331df5-857e-4663-9131-6575bcd3095a\") " pod="openshift-ingress-canary/ingress-canary-2vfjz" Nov 01 06:09:41.092201 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.092102 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:41.092201 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.092149 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mqsk\" (UniqueName: \"kubernetes.io/projected/568704c3-e334-44e1-8373-12a873c305b5-kube-api-access-6mqsk\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:41.092201 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.092177 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568704c3-e334-44e1-8373-12a873c305b5-tmp-dir\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:41.092444 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.092236 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-data-volume\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:41.092444 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.092255 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:41.092444 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.092283 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-crio-socket\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:41.127948 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.127912 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:41.140656 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.140620 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 06:09:41.140656 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.140650 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-h2lht\"" Nov 01 06:09:41.140891 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.140741 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rlspf\"" Nov 01 06:09:41.140891 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.140745 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 06:09:41.140891 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.140745 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 06:09:41.140891 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.140757 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 06:09:41.140891 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.140789 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8r6tp\"" Nov 01 06:09:41.192995 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.192954 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45331df5-857e-4663-9131-6575bcd3095a-cert\") pod \"ingress-canary-2vfjz\" (UID: \"45331df5-857e-4663-9131-6575bcd3095a\") " pod="openshift-ingress-canary/ingress-canary-2vfjz" Nov 01 06:09:41.192995 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.192992 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:41.193238 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.193019 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mqsk\" (UniqueName: \"kubernetes.io/projected/568704c3-e334-44e1-8373-12a873c305b5-kube-api-access-6mqsk\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:41.193238 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.193058 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568704c3-e334-44e1-8373-12a873c305b5-tmp-dir\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:41.193238 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.193097 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-data-volume\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:41.193238 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:41.193118 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 06:09:41.193238 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:41.193187 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls podName:568704c3-e334-44e1-8373-12a873c305b5 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:41.693167505 +0000 UTC m=+32.083352910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls") pod "dns-default-62hgp" (UID: "568704c3-e334-44e1-8373-12a873c305b5") : secret "dns-default-metrics-tls" not found Nov 01 06:09:41.193979 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.193301 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:41.194121 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.194058 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-crio-socket\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:41.194121 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.194102 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xd5q\" (UniqueName: \"kubernetes.io/projected/45331df5-857e-4663-9131-6575bcd3095a-kube-api-access-5xd5q\") pod \"ingress-canary-2vfjz\" (UID: \"45331df5-857e-4663-9131-6575bcd3095a\") " pod="openshift-ingress-canary/ingress-canary-2vfjz" Nov 01 06:09:41.194227 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.193689 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-data-volume\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:41.194227 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.193745 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568704c3-e334-44e1-8373-12a873c305b5-tmp-dir\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:41.194227 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.194134 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/568704c3-e334-44e1-8373-12a873c305b5-config-volume\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:41.194366 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.194224 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-crio-socket\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:41.194366 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:41.193385 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 06:09:41.194366 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.194257 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:41.194366 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:41.194316 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-rbac-proxy-cm podName:36effec9-edbb-40a4-8fe6-bee6b5bde3f1 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:41.694295467 +0000 UTC m=+32.084480880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b9rlh" (UID: "36effec9-edbb-40a4-8fe6-bee6b5bde3f1") : configmap "kube-rbac-proxy" not found Nov 01 06:09:41.194366 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.194362 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5pnm\" (UniqueName: \"kubernetes.io/projected/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-api-access-n5pnm\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:41.194666 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:41.194378 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 06:09:41.194666 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:41.194524 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls podName:36effec9-edbb-40a4-8fe6-bee6b5bde3f1 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:41.694495489 +0000 UTC m=+32.084680894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b9rlh" (UID: "36effec9-edbb-40a4-8fe6-bee6b5bde3f1") : secret "insights-runtime-extractor-tls" not found Nov 01 06:09:41.194666 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.194603 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/568704c3-e334-44e1-8373-12a873c305b5-config-volume\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:41.197467 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.197446 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45331df5-857e-4663-9131-6575bcd3095a-cert\") pod \"ingress-canary-2vfjz\" (UID: \"45331df5-857e-4663-9131-6575bcd3095a\") " pod="openshift-ingress-canary/ingress-canary-2vfjz" Nov 01 06:09:41.252674 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.252599 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mqsk\" (UniqueName: \"kubernetes.io/projected/568704c3-e334-44e1-8373-12a873c305b5-kube-api-access-6mqsk\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:41.254073 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.254052 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5pnm\" (UniqueName: \"kubernetes.io/projected/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-api-access-n5pnm\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:41.254245 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.254227 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xd5q\" (UniqueName: \"kubernetes.io/projected/45331df5-857e-4663-9131-6575bcd3095a-kube-api-access-5xd5q\") pod \"ingress-canary-2vfjz\" (UID: \"45331df5-857e-4663-9131-6575bcd3095a\") " pod="openshift-ingress-canary/ingress-canary-2vfjz" Nov 01 06:09:41.307892 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.307859 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2vfjz" Nov 01 06:09:41.343076 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.343048 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:41.487190 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.487152 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:41.699397 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.699356 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:41.699397 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.699403 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:41.699599 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.699452 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:41.699599 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:41.699504 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 06:09:41.699599 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:41.699542 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 06:09:41.699599 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:41.699569 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 06:09:41.699599 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:41.699572 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls podName:36effec9-edbb-40a4-8fe6-bee6b5bde3f1 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:42.699557105 +0000 UTC m=+33.089742497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b9rlh" (UID: "36effec9-edbb-40a4-8fe6-bee6b5bde3f1") : secret "insights-runtime-extractor-tls" not found Nov 01 06:09:41.699748 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:41.699615 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-rbac-proxy-cm podName:36effec9-edbb-40a4-8fe6-bee6b5bde3f1 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:42.699599017 +0000 UTC m=+33.089784418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b9rlh" (UID: "36effec9-edbb-40a4-8fe6-bee6b5bde3f1") : configmap "kube-rbac-proxy" not found Nov 01 06:09:41.699748 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:41.699633 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls podName:568704c3-e334-44e1-8373-12a873c305b5 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:42.699627384 +0000 UTC m=+33.089812776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls") pod "dns-default-62hgp" (UID: "568704c3-e334-44e1-8373-12a873c305b5") : secret "dns-default-metrics-tls" not found Nov 01 06:09:41.739855 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.739802 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2vfjz"] Nov 01 06:09:41.742925 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:41.742899 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45331df5_857e_4663_9131_6575bcd3095a.slice/crio-0158f578705e45b0aaf2d3520695748637340cc7d13adc72704385867a6eaeed WatchSource:0}: Error finding container 0158f578705e45b0aaf2d3520695748637340cc7d13adc72704385867a6eaeed: Status 404 returned error can't find the container with id 0158f578705e45b0aaf2d3520695748637340cc7d13adc72704385867a6eaeed Nov 01 06:09:41.938883 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.938855 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-ssrz5"] Nov 01 06:09:41.982507 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.982439 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-ssrz5" Nov 01 06:09:41.997731 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:41.997697 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-ssrz5"] Nov 01 06:09:42.034856 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.034801 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 01 06:09:42.034856 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.034826 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 01 06:09:42.035084 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.034969 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 01 06:09:42.035084 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.034986 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jtkl2\"" Nov 01 06:09:42.035084 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.035022 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 01 06:09:42.102601 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.102559 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ff209546-604b-462a-8f67-dee8fc8698d3-signing-cabundle\") pod \"service-ca-5fffc97bdf-ssrz5\" (UID: \"ff209546-604b-462a-8f67-dee8fc8698d3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ssrz5" Nov 01 06:09:42.103029 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.102607 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ff209546-604b-462a-8f67-dee8fc8698d3-signing-key\") pod \"service-ca-5fffc97bdf-ssrz5\" (UID: \"ff209546-604b-462a-8f67-dee8fc8698d3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ssrz5" Nov 01 06:09:42.103029 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.102633 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57g2w\" (UniqueName: \"kubernetes.io/projected/ff209546-604b-462a-8f67-dee8fc8698d3-kube-api-access-57g2w\") pod \"service-ca-5fffc97bdf-ssrz5\" (UID: \"ff209546-604b-462a-8f67-dee8fc8698d3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ssrz5" Nov 01 06:09:42.146321 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.146294 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:42.203963 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.203932 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ff209546-604b-462a-8f67-dee8fc8698d3-signing-cabundle\") pod \"service-ca-5fffc97bdf-ssrz5\" (UID: \"ff209546-604b-462a-8f67-dee8fc8698d3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ssrz5" Nov 01 06:09:42.204091 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.203980 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ff209546-604b-462a-8f67-dee8fc8698d3-signing-key\") pod \"service-ca-5fffc97bdf-ssrz5\" (UID: \"ff209546-604b-462a-8f67-dee8fc8698d3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ssrz5" Nov 01 06:09:42.204155 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.204094 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57g2w\" (UniqueName: \"kubernetes.io/projected/ff209546-604b-462a-8f67-dee8fc8698d3-kube-api-access-57g2w\") pod \"service-ca-5fffc97bdf-ssrz5\" (UID: \"ff209546-604b-462a-8f67-dee8fc8698d3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ssrz5" Nov 01 06:09:42.204573 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.204554 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ff209546-604b-462a-8f67-dee8fc8698d3-signing-cabundle\") pod \"service-ca-5fffc97bdf-ssrz5\" (UID: \"ff209546-604b-462a-8f67-dee8fc8698d3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ssrz5" Nov 01 06:09:42.206451 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.206420 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ff209546-604b-462a-8f67-dee8fc8698d3-signing-key\") pod \"service-ca-5fffc97bdf-ssrz5\" (UID: \"ff209546-604b-462a-8f67-dee8fc8698d3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ssrz5" Nov 01 06:09:42.265275 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.265204 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57g2w\" (UniqueName: \"kubernetes.io/projected/ff209546-604b-462a-8f67-dee8fc8698d3-kube-api-access-57g2w\") pod \"service-ca-5fffc97bdf-ssrz5\" (UID: \"ff209546-604b-462a-8f67-dee8fc8698d3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ssrz5" Nov 01 06:09:42.292457 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.292424 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:42.294037 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.294010 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2vfjz" event={"ID":"45331df5-857e-4663-9131-6575bcd3095a","Type":"ContainerStarted","Data":"0158f578705e45b0aaf2d3520695748637340cc7d13adc72704385867a6eaeed"} Nov 01 06:09:42.304414 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.304390 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-ssrz5" Nov 01 06:09:42.687609 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.687570 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-ssrz5"] Nov 01 06:09:42.691699 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:42.691657 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff209546_604b_462a_8f67_dee8fc8698d3.slice/crio-fa0cb52b5dbb3cb0c44446945e595216808c2b27c8f78dd6a11506b57c09598a WatchSource:0}: Error finding container fa0cb52b5dbb3cb0c44446945e595216808c2b27c8f78dd6a11506b57c09598a: Status 404 returned error can't find the container with id fa0cb52b5dbb3cb0c44446945e595216808c2b27c8f78dd6a11506b57c09598a Nov 01 06:09:42.708099 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.708071 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:42.708237 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.708134 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:42.708237 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.708168 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:42.708237 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:42.708205 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 06:09:42.708372 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:42.708271 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 06:09:42.708372 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:42.708275 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 06:09:42.708372 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:42.708284 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-rbac-proxy-cm podName:36effec9-edbb-40a4-8fe6-bee6b5bde3f1 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:44.708260238 +0000 UTC m=+35.098445654 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b9rlh" (UID: "36effec9-edbb-40a4-8fe6-bee6b5bde3f1") : configmap "kube-rbac-proxy" not found Nov 01 06:09:42.708372 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:42.708320 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls podName:568704c3-e334-44e1-8373-12a873c305b5 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:44.708306128 +0000 UTC m=+35.098491552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls") pod "dns-default-62hgp" (UID: "568704c3-e334-44e1-8373-12a873c305b5") : secret "dns-default-metrics-tls" not found Nov 01 06:09:42.708372 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:42.708335 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls podName:36effec9-edbb-40a4-8fe6-bee6b5bde3f1 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:44.708328336 +0000 UTC m=+35.098513727 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b9rlh" (UID: "36effec9-edbb-40a4-8fe6-bee6b5bde3f1") : secret "insights-runtime-extractor-tls" not found Nov 01 06:09:42.725224 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.725193 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:42.975492 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:42.975421 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:43.010554 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:43.010518 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs\") pod \"network-metrics-daemon-b98x2\" (UID: \"8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8\") " pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:43.010702 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:43.010561 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:43.010702 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:43.010678 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 06:09:43.010791 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:43.010746 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs podName:8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8 nodeName:}" failed. No retries permitted until 2026-11-01 06:10:15.010731912 +0000 UTC m=+65.400917317 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs") pod "network-metrics-daemon-b98x2" (UID: "8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8") : secret "metrics-daemon-secret" not found Nov 01 06:09:43.012915 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:43.012892 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:43.111228 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:43.111189 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wqjh\" (UniqueName: \"kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh\") pod \"network-check-target-vvpd2\" (UID: \"6228c388-659f-4ca7-8d51-4fa5658424e2\") " pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:43.113776 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:43.113757 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wqjh\" (UniqueName: \"kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh\") pod \"network-check-target-vvpd2\" (UID: \"6228c388-659f-4ca7-8d51-4fa5658424e2\") " pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:43.130943 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:43.130913 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:43.196846 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:43.196801 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:43.201989 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:43.201959 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:43.307056 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:43.307022 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-ssrz5" event={"ID":"ff209546-604b-462a-8f67-dee8fc8698d3","Type":"ContainerStarted","Data":"fa0cb52b5dbb3cb0c44446945e595216808c2b27c8f78dd6a11506b57c09598a"} Nov 01 06:09:43.587051 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:43.587022 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-txvqp"] Nov 01 06:09:43.590261 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:43.590235 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0855c2_93a3_44a5_9311_63d891705e0e.slice/crio-99ce3c1f6bf83c7eb09e0ae7bd283e9227842a2428109c76d9dbacfa335d43c8 WatchSource:0}: Error finding container 99ce3c1f6bf83c7eb09e0ae7bd283e9227842a2428109c76d9dbacfa335d43c8: Status 404 returned error can't find the container with id 99ce3c1f6bf83c7eb09e0ae7bd283e9227842a2428109c76d9dbacfa335d43c8 Nov 01 06:09:43.638017 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:43.637887 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vvpd2"] Nov 01 06:09:43.640634 ip-10-0-140-68 kubenswrapper[2567]: W1101 06:09:43.640605 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6228c388_659f_4ca7_8d51_4fa5658424e2.slice/crio-ddf6129b10d2dd9dc196acb809707ed5f454cb9b5d2e92f3e0cbc7854c0bd333 WatchSource:0}: Error finding container ddf6129b10d2dd9dc196acb809707ed5f454cb9b5d2e92f3e0cbc7854c0bd333: Status 404 returned error can't find the container with id ddf6129b10d2dd9dc196acb809707ed5f454cb9b5d2e92f3e0cbc7854c0bd333 Nov 01 06:09:44.312940 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:44.312897 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vvpd2" event={"ID":"6228c388-659f-4ca7-8d51-4fa5658424e2","Type":"ContainerStarted","Data":"ddf6129b10d2dd9dc196acb809707ed5f454cb9b5d2e92f3e0cbc7854c0bd333"} Nov 01 06:09:44.314370 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:44.314336 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-txvqp" event={"ID":"5b0855c2-93a3-44a5-9311-63d891705e0e","Type":"ContainerStarted","Data":"99ce3c1f6bf83c7eb09e0ae7bd283e9227842a2428109c76d9dbacfa335d43c8"} Nov 01 06:09:44.725043 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:44.725002 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:44.725209 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:44.725073 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:44.725209 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:44.725128 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:44.725329 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:44.725242 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 06:09:44.725329 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:44.725305 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-rbac-proxy-cm podName:36effec9-edbb-40a4-8fe6-bee6b5bde3f1 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:48.725286721 +0000 UTC m=+39.115472128 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b9rlh" (UID: "36effec9-edbb-40a4-8fe6-bee6b5bde3f1") : configmap "kube-rbac-proxy" not found Nov 01 06:09:44.725729 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:44.725710 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 06:09:44.725809 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:44.725760 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls podName:36effec9-edbb-40a4-8fe6-bee6b5bde3f1 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:48.725745897 +0000 UTC m=+39.115931302 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b9rlh" (UID: "36effec9-edbb-40a4-8fe6-bee6b5bde3f1") : secret "insights-runtime-extractor-tls" not found Nov 01 06:09:44.725910 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:44.725818 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 06:09:44.725910 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:44.725863 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls podName:568704c3-e334-44e1-8373-12a873c305b5 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:48.725852082 +0000 UTC m=+39.116037488 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls") pod "dns-default-62hgp" (UID: "568704c3-e334-44e1-8373-12a873c305b5") : secret "dns-default-metrics-tls" not found Nov 01 06:09:45.791108 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:45.791003 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:46.326239 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:46.326185 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-ssrz5" event={"ID":"ff209546-604b-462a-8f67-dee8fc8698d3","Type":"ContainerStarted","Data":"a2cebeb3910bb40f543735a638403242cec96932d427bd47fca80cd58c4e02d6"} Nov 01 06:09:46.440859 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:46.440728 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-ssrz5" podStartSLOduration=1.97917267 podStartE2EDuration="5.440708881s" totalImagesPullingTime="3.461536211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:46.439691808 +0000 UTC m=+36.829877223" watchObservedRunningTime="2026-11-01 06:09:46.440708881 +0000 UTC m=+36.830894297" Nov 01 06:09:46.827697 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:46.827664 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ks27g_939039b6-e071-4a27-b24e-20bb2669dc15/dns-node-resolver/0.log" Nov 01 06:09:47.332699 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:47.332618 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2vfjz" event={"ID":"45331df5-857e-4663-9131-6575bcd3095a","Type":"ContainerStarted","Data":"fc10af097c1722a5a5d85f7543b6a2c89ddd182c05e2c48e88c78cee7e835e47"} Nov 01 06:09:47.445378 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:47.445311 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2vfjz" podStartSLOduration=2.106832916 podStartE2EDuration="7.445295146s" totalImagesPullingTime="5.33846223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:47.44455245 +0000 UTC m=+37.834737908" watchObservedRunningTime="2026-11-01 06:09:47.445295146 +0000 UTC m=+37.835480616" Nov 01 06:09:47.493899 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:47.493866 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:47.643890 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:47.643796 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:48.049008 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:48.048922 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:48.202795 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:48.202729 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:48.226950 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:48.226923 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vth7q_7b856c58-9b9e-47b3-947a-ed56ae892e9b/node-ca/0.log" Nov 01 06:09:48.383278 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:48.383245 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:48.762462 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:48.762368 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:48.762462 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:48.762443 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:48.762728 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:48.762484 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:48.762728 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:48.762594 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 06:09:48.762728 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:48.762669 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls podName:568704c3-e334-44e1-8373-12a873c305b5 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:56.762651279 +0000 UTC m=+47.152836689 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls") pod "dns-default-62hgp" (UID: "568704c3-e334-44e1-8373-12a873c305b5") : secret "dns-default-metrics-tls" not found Nov 01 06:09:48.762728 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:48.762667 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 06:09:48.762957 ip-10-0-140-68 kubenswrapper[2567]: E1101 06:09:48.762757 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls podName:36effec9-edbb-40a4-8fe6-bee6b5bde3f1 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:56.76273606 +0000 UTC m=+47.152921477 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b9rlh" (UID: "36effec9-edbb-40a4-8fe6-bee6b5bde3f1") : secret "insights-runtime-extractor-tls" not found Nov 01 06:09:48.763055 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:48.763032 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:48.826307 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:48.826268 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2vfjz_45331df5-857e-4663-9131-6575bcd3095a/serve-healthcheck-canary/0.log" Nov 01 06:09:49.201059 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:49.201014 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:49.339291 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:49.339239 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vvpd2" event={"ID":"6228c388-659f-4ca7-8d51-4fa5658424e2","Type":"ContainerStarted","Data":"04c6b103bc0037bf3c9d7288c405c915f49a6038dcd08454ad14e15a4fb4810c"} Nov 01 06:09:49.339467 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:49.339365 2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:49.352557 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:49.352522 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:49.453237 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:49.453137 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vvpd2" podStartSLOduration=34.697172385 podStartE2EDuration="39.453124504s" totalImagesPullingTime="4.755952119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:49.452680588 +0000 UTC m=+39.842866003" watchObservedRunningTime="2026-11-01 06:09:49.453124504 +0000 UTC m=+39.843309917" Nov 01 06:09:49.494850 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:49.494802 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:49.701703 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:49.701656 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:49.897489 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:49.897458 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:50.097813 ip-10-0-140-68 kubenswrapper[2567]: I1101 06:09:50.097759 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:50.182415 ip-10-0-140-68 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 06:09:50.193221 ip-10-0-140-68 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 06:09:50.193394 ip-10-0-140-68 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 06:09:50.193445 ip-10-0-140-68 systemd[1]: kubelet.service: Consumed 2.569s CPU time. Nov 01 06:09:50.204286 ip-10-0-140-68 systemd[1]: Starting Kubernetes Kubelet... Nov 01 06:09:50.297813 ip-10-0-140-68 kubenswrapper[4385]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 06:09:50.297813 ip-10-0-140-68 kubenswrapper[4385]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 06:09:50.297813 ip-10-0-140-68 kubenswrapper[4385]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 06:09:50.297813 ip-10-0-140-68 kubenswrapper[4385]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 06:09:50.300282 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300189 4385 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:50.300282 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300279 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:50.300282 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300284 4385 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:50.300282 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300289 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:50.300566 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300293 4385 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:50.300566 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300297 4385 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:50.300566 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300301 4385 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:50.300566 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300305 4385 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:50.300566 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300309 4385 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:50.300566 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300313 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:50.300566 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300316 4385 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:50.300566 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300320 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:50.300566 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300324 4385 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:50.300566 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300328 4385 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:50.300566 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300332 4385 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:50.300566 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300341 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:50.300566 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300345 4385 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:50.300566 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300350 4385 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:50.300566 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300354 4385 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:50.300566 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300358 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:50.300566 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300362 4385 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:50.300566 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300366 4385 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:50.300566 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300370 4385 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:50.300566 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300375 4385 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:50.301449 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300378 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:50.301449 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300382 4385 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:50.301449 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300386 4385 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:50.301449 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300390 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:50.301449 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300395 4385 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:50.301449 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300398 4385 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:50.301449 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300402 4385 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:50.301449 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300406 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:50.301449 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300410 4385 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:50.301449 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300414 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:50.301449 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300418 4385 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:50.301449 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300422 4385 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:50.301449 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300427 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:50.301449 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300431 4385 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:50.301449 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300436 4385 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:50.301449 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300441 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:50.301449 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300445 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:50.301449 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300450 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:50.301449 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300454 4385 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:50.302305 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300458 4385 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:50.302305 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300463 4385 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:50.302305 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300467 4385 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:50.302305 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300471 4385 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:50.302305 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300476 4385 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:50.302305 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300480 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:50.302305 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300485 4385 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:50.302305 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300489 4385 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:50.302305 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300497 4385 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:50.302305 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300505 4385 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:50.302305 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300510 4385 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:50.302305 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300515 4385 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:50.302305 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300519 4385 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:50.302305 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300523 4385 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:50.302305 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300528 4385 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:50.302305 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300534 4385 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:50.302305 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300538 4385 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:50.302305 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300543 4385 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:50.302305 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300547 4385 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:50.303158 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300551 4385 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:50.303158 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300555 4385 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:50.303158 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300560 4385 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:50.303158 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300564 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:50.303158 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300568 4385 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:50.303158 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300572 4385 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:50.303158 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300577 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:50.303158 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300581 4385 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:50.303158 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300585 4385 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:50.303158 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300589 4385 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:50.303158 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300593 4385 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:50.303158 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300597 4385 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:50.303158 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300602 4385 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:50.303158 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300606 4385 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:50.303158 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300610 4385 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:50.303158 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300615 4385 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:50.303158 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300623 4385 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:50.303158 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300629 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:50.303158 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300635 4385 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:50.304028 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300640 4385 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:50.304028 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300646 4385 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:50.304028 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300655 4385 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:50.304028 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300660 4385 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:50.304028 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300664 4385 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:50.304028 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300669 4385 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:50.304028 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300673 4385 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:50.304028 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300677 4385 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:50.304028 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300681 4385 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:50.304028 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300686 4385 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:50.304028 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300691 4385 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:50.304028 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300695 4385 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:50.304028 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300700 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:50.304028 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300704 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:50.304028 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300709 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:50.304028 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300713 4385 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:50.304028 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300717 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:50.304028 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300721 4385 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:50.304028 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300726 4385 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:50.304899 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300730 4385 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:50.304899 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300734 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:50.304899 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300739 4385 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:50.304899 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300743 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:50.304899 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300747 4385 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:50.304899 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.300752 4385 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:50.304899 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302168 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:50.304899 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302179 4385 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:50.304899 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302184 4385 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:50.304899 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302189 4385 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:50.304899 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302193 4385 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:50.304899 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302199 4385 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:50.304899 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302204 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:50.304899 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302208 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:50.304899 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302213 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:50.304899 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302221 4385 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:50.304899 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302225 4385 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:50.304899 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302230 4385 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:50.304899 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302234 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:50.304899 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302239 4385 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:50.305790 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302243 4385 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:50.305790 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302247 4385 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:50.305790 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302253 4385 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:50.305790 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302258 4385 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:50.305790 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302265 4385 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:50.305790 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302271 4385 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:50.305790 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302276 4385 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:50.305790 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302281 4385 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:50.305790 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302285 4385 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:50.305790 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302290 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:50.305790 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302294 4385 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:50.305790 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302299 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:50.305790 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302303 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:50.305790 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302306 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:50.305790 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302311 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:50.305790 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302314 4385 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:50.305790 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302318 4385 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:50.305790 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302322 4385 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:50.305790 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302327 4385 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:50.305790 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302331 4385 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:50.306666 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302335 4385 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:50.306666 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302340 4385 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:50.306666 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302344 4385 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:50.306666 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302349 4385 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:50.306666 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302354 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:50.306666 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302359 4385 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:50.306666 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302363 4385 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:50.306666 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302370 4385 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:50.306666 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302374 4385 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:50.306666 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302378 4385 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:50.306666 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302383 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:50.306666 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302387 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:50.306666 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302391 4385 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:50.306666 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302396 4385 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:50.306666 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302400 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:50.306666 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302404 4385 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:50.306666 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302408 4385 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:50.306666 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302413 4385 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:50.306666 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302417 4385 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:50.306666 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302421 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:50.307365 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302426 4385 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:50.307365 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302430 4385 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:50.307365 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302434 4385 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:50.307365 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302438 4385 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:50.307365 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302443 4385 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:50.307365 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302447 4385 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:50.307365 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302451 4385 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:50.307365 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302455 4385 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:50.307365 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302459 4385 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:50.307365 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302463 4385 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:50.307365 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302468 4385 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:50.307365 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302473 4385 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:50.307365 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302477 4385 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:50.307365 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302481 4385 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:50.307365 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302486 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:50.307365 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302491 4385 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:50.307365 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302496 4385 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:50.307365 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302501 4385 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:50.307365 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302505 4385 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:50.307868 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302515 4385 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:50.307868 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302519 4385 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:50.307868 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302524 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:50.307868 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302528 4385 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:50.307868 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302532 4385 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:50.307868 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302537 4385 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:50.307868 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302541 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:50.307868 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302545 4385 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:50.307868 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302550 4385 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:50.307868 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302554 4385 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:50.307868 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302558 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:50.307868 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302563 4385 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:50.307868 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302567 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:50.307868 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302572 4385 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:50.307868 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302576 4385 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:50.307868 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302581 4385 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:50.307868 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302585 4385 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:50.307868 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302590 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:50.307868 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302594 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:50.308475 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302598 4385 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:50.308475 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302602 4385 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:50.308475 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302606 4385 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:50.308475 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302611 4385 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:50.308475 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302615 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:50.308475 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302620 4385 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:50.308475 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302624 4385 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:50.308475 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302629 4385 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:50.308475 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302633 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:50.308475 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302637 4385 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:50.308475 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302641 4385 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:50.308475 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302645 4385 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:50.308475 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302650 4385 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:50.308475 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.302660 4385 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:50.308475 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303515 4385 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:50.308475 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303526 4385 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:50.308475 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303530 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:50.308475 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303537 4385 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:50.308475 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303544 4385 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:50.309014 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303548 4385 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:50.309014 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303552 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:50.309014 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303557 4385 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:50.309014 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303562 4385 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:50.309014 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303566 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:50.309014 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303570 4385 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:50.309014 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303575 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:50.309014 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303579 4385 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:50.309014 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303584 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:50.309014 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303588 4385 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:50.309014 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303592 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:50.309014 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303596 4385 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:50.309014 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303601 4385 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:50.309014 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303605 4385 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:50.309014 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303610 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:50.309014 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303614 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:50.309014 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303618 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:50.309014 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303623 4385 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:50.309014 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303627 4385 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:50.309014 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303631 4385 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:50.309521 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303635 4385 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:50.309521 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303639 4385 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:50.309521 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303643 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:50.309521 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303648 4385 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:50.309521 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303652 4385 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:50.309521 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303657 4385 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:50.309521 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303664 4385 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:50.309521 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303668 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:50.309521 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303673 4385 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:50.309521 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303677 4385 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:50.309521 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303681 4385 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:50.309521 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303686 4385 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:50.309521 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303690 4385 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:50.309521 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303694 4385 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:50.309521 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303699 4385 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:50.309521 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303703 4385 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:50.309521 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303707 4385 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:50.309521 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303711 4385 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:50.309521 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303716 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:50.309521 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303722 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:50.310035 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303726 4385 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:50.310035 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303731 4385 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:50.310035 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303735 4385 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:50.310035 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303739 4385 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:50.310035 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303743 4385 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:50.310035 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303748 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:50.310035 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303752 4385 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:50.310035 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303756 4385 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:50.310035 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303760 4385 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:50.310035 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303764 4385 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:50.310035 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303769 4385 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:50.310035 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303773 4385 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:50.310035 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303777 4385 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:50.310035 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303781 4385 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:50.310035 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303785 4385 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:50.310035 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303789 4385 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:50.310035 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303794 4385 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:50.310035 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303798 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:50.310035 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303805 4385 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:50.310035 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303810 4385 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:50.310532 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303814 4385 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:50.310532 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303818 4385 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:50.310532 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303822 4385 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:50.310532 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303826 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:50.310532 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303848 4385 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:50.310532 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303852 4385 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:50.310532 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303856 4385 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:50.310532 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303860 4385 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:50.310532 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303865 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:50.310532 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303869 4385 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:50.310532 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303875 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:50.310532 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303882 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:50.310532 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303886 4385 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:50.310532 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303890 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:50.310532 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303894 4385 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:50.310532 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303898 4385 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:50.310532 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303903 4385 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:50.310532 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303908 4385 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:50.310532 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303912 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:50.311012 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303917 4385 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:50.311012 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303921 4385 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:50.311012 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303925 4385 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:50.311012 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303929 4385 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:50.311012 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303934 4385 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:50.311012 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303938 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:50.311012 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303943 4385 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:50.311012 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303947 4385 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:50.311012 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303952 4385 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:50.311012 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303956 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:50.311012 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303960 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:50.311012 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303966 4385 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:50.311012 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303971 4385 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:50.311012 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303976 4385 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:50.311012 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303980 4385 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:50.311012 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303984 4385 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:50.311012 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303988 4385 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:50.311012 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303992 4385 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:50.311012 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.303997 4385 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:50.311012 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.304001 4385 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.304010 4385 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.304015 4385 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304161 4385 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304173 4385 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304188 4385 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304196 4385 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304204 4385 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304209 4385 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304216 4385 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304222 4385 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304228 4385 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304232 4385 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304239 4385 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304244 4385 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304249 4385 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304254 4385 flags.go:64] FLAG: --cgroup-root="" Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304259 4385 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304263 4385 flags.go:64] FLAG: --client-ca-file="" Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304268 4385 flags.go:64] FLAG: --cloud-provider="external" Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304273 4385 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304280 4385 flags.go:64] FLAG: --cluster-domain="" Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304285 4385 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 06:09:50.311495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304290 4385 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304295 4385 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304301 4385 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304308 4385 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304313 4385 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304318 4385 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304323 4385 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304328 4385 flags.go:64] FLAG: --contention-profiling="false" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304333 4385 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304338 4385 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304343 4385 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304348 4385 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304355 4385 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304360 4385 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304366 4385 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304372 4385 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304377 4385 flags.go:64] FLAG: --enable-server="true" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304382 4385 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304389 4385 flags.go:64] FLAG: --event-burst="100" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304394 4385 flags.go:64] FLAG: --event-qps="50" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304399 4385 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304404 4385 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304409 4385 flags.go:64] FLAG: --eviction-hard="" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304416 4385 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 06:09:50.312063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304421 4385 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304426 4385 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304431 4385 flags.go:64] FLAG: --eviction-soft="" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304436 4385 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304441 4385 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304446 4385 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304451 4385 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304456 4385 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304461 4385 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304465 4385 flags.go:64] FLAG: --feature-gates="" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304472 4385 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304477 4385 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304483 4385 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304489 4385 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304494 4385 flags.go:64] FLAG: --healthz-port="10248" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304499 4385 flags.go:64] FLAG: --help="false" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304504 4385 flags.go:64] FLAG: --hostname-override="ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304509 4385 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304514 4385 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304519 4385 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304525 4385 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304530 4385 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304537 4385 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 06:09:50.312675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304541 4385 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304546 4385 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304550 4385 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304555 4385 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304560 4385 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304565 4385 flags.go:64] FLAG: --kube-reserved="" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304570 4385 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304575 4385 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304580 4385 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304585 4385 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304590 4385 flags.go:64] FLAG: --lock-file="" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304594 4385 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304599 4385 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304604 4385 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304613 4385 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304618 4385 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304623 4385 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304628 4385 flags.go:64] FLAG: --logging-format="text" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304633 4385 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304639 4385 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304644 4385 flags.go:64] FLAG: --manifest-url="" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304648 4385 flags.go:64] FLAG: --manifest-url-header="" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304656 4385 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304661 4385 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304667 4385 flags.go:64] FLAG: --max-pods="110" Nov 01 06:09:50.313257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304672 4385 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304677 4385 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304682 4385 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304686 4385 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304691 4385 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304696 4385 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304702 4385 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304716 4385 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304721 4385 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304726 4385 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304731 4385 flags.go:64] FLAG: --pod-cidr="" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304735 4385 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304740 4385 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304745 4385 flags.go:64] FLAG: --pods-per-core="0" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304749 4385 flags.go:64] FLAG: --port="10250" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304755 4385 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304759 4385 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-080efc09aabf5cbfe" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304765 4385 flags.go:64] FLAG: --qos-reserved="" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304769 4385 flags.go:64] FLAG: --read-only-port="10255" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304774 4385 flags.go:64] FLAG: --register-node="true" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304780 4385 flags.go:64] FLAG: --register-with-taints="" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304786 4385 flags.go:64] FLAG: --registry-burst="10" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304790 4385 flags.go:64] FLAG: --registry-qps="5" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304795 4385 flags.go:64] FLAG: --reserved-cpus="" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304800 4385 flags.go:64] FLAG: --reserved-memory="" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304806 4385 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 06:09:50.313875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304811 4385 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304816 4385 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304820 4385 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304825 4385 flags.go:64] FLAG: --runonce="false" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304848 4385 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304854 4385 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304859 4385 flags.go:64] FLAG: --seccomp-default="false" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304863 4385 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304867 4385 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304872 4385 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304877 4385 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304882 4385 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304888 4385 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304893 4385 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304898 4385 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304902 4385 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304907 4385 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304911 4385 flags.go:64] FLAG: --system-cgroups="" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304915 4385 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304924 4385 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304930 4385 flags.go:64] FLAG: --tls-cert-file="" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304935 4385 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304942 4385 flags.go:64] FLAG: --tls-min-version="" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304947 4385 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304951 4385 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 06:09:50.314503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304956 4385 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 06:09:50.315135 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304961 4385 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 06:09:50.315135 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304966 4385 flags.go:64] FLAG: --v="2" Nov 01 06:09:50.315135 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304973 4385 flags.go:64] FLAG: --version="false" Nov 01 06:09:50.315135 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304980 4385 flags.go:64] FLAG: --vmodule="" Nov 01 06:09:50.315135 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304986 4385 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 06:09:50.315135 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.304992 4385 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 06:09:50.315135 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305210 4385 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:50.315135 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305217 4385 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:50.315135 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305223 4385 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:50.315135 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305228 4385 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:50.315135 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305233 4385 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:50.315135 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305238 4385 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:50.315135 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305242 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:50.315135 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305247 4385 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:50.315135 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305252 4385 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:50.315135 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305256 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:50.315135 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305260 4385 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:50.315135 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305265 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:50.315135 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305271 4385 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:50.315135 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305276 4385 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:50.315135 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305280 4385 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:50.315996 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305284 4385 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:50.315996 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305289 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:50.315996 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305293 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:50.315996 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305298 4385 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:50.315996 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305302 4385 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:50.315996 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305309 4385 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:50.315996 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305313 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:50.315996 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305318 4385 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:50.315996 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305322 4385 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:50.315996 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305326 4385 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:50.315996 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305330 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:50.315996 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305335 4385 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:50.315996 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305339 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:50.315996 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305343 4385 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:50.315996 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305348 4385 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:50.315996 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305352 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:50.315996 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305356 4385 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:50.315996 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305360 4385 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:50.315996 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305365 4385 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:50.315996 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305369 4385 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:50.316764 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305373 4385 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:50.316764 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305378 4385 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:50.316764 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305382 4385 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:50.316764 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305387 4385 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:50.316764 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305392 4385 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:50.316764 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305396 4385 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:50.316764 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305400 4385 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:50.316764 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305406 4385 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:50.316764 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305410 4385 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:50.316764 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305417 4385 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:50.316764 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305421 4385 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:50.316764 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305425 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:50.316764 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305430 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:50.316764 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305434 4385 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:50.316764 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305438 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:50.316764 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305442 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:50.316764 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305447 4385 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:50.316764 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305453 4385 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:50.316764 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305457 4385 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:50.316764 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305461 4385 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:50.317436 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305466 4385 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:50.317436 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305470 4385 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:50.317436 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305474 4385 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:50.317436 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305478 4385 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:50.317436 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305482 4385 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:50.317436 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305487 4385 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:50.317436 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305492 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:50.317436 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305496 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:50.317436 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305500 4385 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:50.317436 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305504 4385 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:50.317436 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305509 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:50.317436 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305513 4385 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:50.317436 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305517 4385 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:50.317436 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305521 4385 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:50.317436 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305528 4385 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:50.317436 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305534 4385 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:50.317436 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305541 4385 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:50.317436 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305545 4385 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:50.317436 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305549 4385 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:50.318086 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305554 4385 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:50.318086 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305558 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:50.318086 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305564 4385 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:50.318086 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305569 4385 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:50.318086 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305573 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:50.318086 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305577 4385 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:50.318086 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305582 4385 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:50.318086 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305587 4385 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:50.318086 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305592 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:50.318086 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305596 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:50.318086 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305602 4385 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:50.318086 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305606 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:50.318086 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305611 4385 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:50.318086 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305615 4385 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:50.318086 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305619 4385 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:50.318086 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305623 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:50.318086 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305628 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:50.318086 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305632 4385 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:50.318086 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305636 4385 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:50.318652 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305641 4385 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:50.318652 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305646 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:50.318652 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305650 4385 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:50.318652 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305654 4385 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:50.318652 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305658 4385 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:50.318652 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305665 4385 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:50.318652 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305671 4385 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:50.318652 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305677 4385 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:50.318652 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305682 4385 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:50.318652 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305687 4385 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:50.318652 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305692 4385 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:50.318652 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305697 4385 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:50.318652 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305701 4385 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:50.318652 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.305743 4385 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 06:09:50.318652 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305826 4385 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:50.319127 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305856 4385 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:50.319127 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305865 4385 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:50.319127 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305870 4385 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:50.319127 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305875 4385 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:50.319127 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305880 4385 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:50.319127 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305885 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:50.319127 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305890 4385 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:50.319127 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305894 4385 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:50.319127 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305901 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:50.319127 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305906 4385 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:50.319127 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305910 4385 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:50.319127 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305915 4385 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:50.319127 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305919 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:50.319127 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305924 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:50.319127 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305928 4385 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:50.319127 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305933 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:50.319127 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305937 4385 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:50.319127 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305942 4385 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:50.319127 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305946 4385 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:50.320042 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305950 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:50.320042 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305955 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:50.320042 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305959 4385 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:50.320042 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305964 4385 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:50.320042 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305968 4385 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:50.320042 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305972 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:50.320042 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305977 4385 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:50.320042 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305988 4385 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:50.320042 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.305993 4385 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:50.320042 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306000 4385 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:50.320042 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306007 4385 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:50.320042 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306012 4385 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:50.320042 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306016 4385 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:50.320042 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306021 4385 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:50.320042 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306029 4385 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:50.320042 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306033 4385 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:50.320042 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306037 4385 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:50.320042 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306042 4385 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:50.320042 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306046 4385 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:50.320515 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306050 4385 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:50.320515 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306054 4385 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:50.320515 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306060 4385 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:50.320515 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306065 4385 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:50.320515 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306069 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:50.320515 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306073 4385 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:50.320515 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306078 4385 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:50.320515 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306082 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:50.320515 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306087 4385 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:50.320515 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306091 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:50.320515 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306095 4385 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:50.320515 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306099 4385 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:50.320515 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306103 4385 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:50.320515 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306108 4385 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:50.320515 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306112 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:50.320515 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306117 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:50.320515 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306122 4385 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:50.320515 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306127 4385 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:50.320515 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306131 4385 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:50.320515 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306135 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:50.321078 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306140 4385 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:50.321078 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306144 4385 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:50.321078 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306148 4385 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:50.321078 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306152 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:50.321078 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306158 4385 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:50.321078 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306162 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:50.321078 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306166 4385 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:50.321078 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306174 4385 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:50.321078 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306178 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:50.321078 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306183 4385 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:50.321078 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306187 4385 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:50.321078 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306191 4385 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:50.321078 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306195 4385 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:50.321078 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306200 4385 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:50.321078 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306207 4385 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:50.321078 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306211 4385 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:50.321078 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306215 4385 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:50.321078 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306220 4385 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:50.321078 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306224 4385 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:50.321078 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306229 4385 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:50.321817 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306235 4385 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:50.321817 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306239 4385 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:50.321817 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306244 4385 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:50.321817 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306249 4385 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:50.321817 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306253 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:50.321817 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306257 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:50.321817 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306262 4385 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:50.321817 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306266 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:50.321817 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306270 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:50.321817 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306274 4385 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:50.321817 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306278 4385 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:50.321817 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306283 4385 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:50.321817 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306287 4385 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:50.321817 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306292 4385 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:50.321817 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306296 4385 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:50.321817 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306300 4385 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:50.321817 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306306 4385 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:50.321817 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306311 4385 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:50.321817 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306317 4385 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:50.321817 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306323 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:50.322584 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306328 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:50.322584 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306332 4385 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:50.322584 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306336 4385 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:50.322584 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306341 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:50.322584 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306345 4385 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:50.322584 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306349 4385 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:50.322584 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.306355 4385 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:50.322584 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.311170 4385 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 06:09:50.322584 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.311187 4385 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 06:09:50.322584 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.311205 4385 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 06:09:50.322584 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.311210 4385 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 06:09:50.322584 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311285 4385 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:50.322584 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311290 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:50.322584 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311294 4385 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:50.322584 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311297 4385 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:50.322584 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311300 4385 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:50.322584 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311302 4385 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:50.322584 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311305 4385 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:50.322584 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311308 4385 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:50.323083 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311310 4385 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:50.323083 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311313 4385 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:50.323083 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311316 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:50.323083 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311319 4385 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:50.323083 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311322 4385 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:50.323083 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311325 4385 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:50.323083 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311328 4385 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:50.323083 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311331 4385 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:50.323083 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311334 4385 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:50.323083 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311337 4385 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:50.323083 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311340 4385 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:50.323083 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311343 4385 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:50.323083 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311345 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:50.323083 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311348 4385 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:50.323083 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311351 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:50.323083 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311354 4385 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:50.323083 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311356 4385 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:50.323083 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311360 4385 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:50.323083 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311363 4385 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:50.323582 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311366 4385 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:50.323582 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311368 4385 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:50.323582 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311378 4385 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:50.323582 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311381 4385 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:50.323582 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311384 4385 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:50.323582 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311387 4385 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:50.323582 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311389 4385 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:50.323582 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311392 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:50.323582 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311395 4385 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:50.323582 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311397 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:50.323582 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311400 4385 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:50.323582 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311403 4385 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:50.323582 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311406 4385 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:50.323582 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311409 4385 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:50.323582 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311411 4385 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:50.323582 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311414 4385 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:50.323582 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311417 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:50.323582 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311419 4385 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:50.323582 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311422 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:50.323582 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311425 4385 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:50.324096 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311427 4385 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:50.324096 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311430 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:50.324096 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311433 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:50.324096 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311436 4385 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:50.324096 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311439 4385 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:50.324096 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311441 4385 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:50.324096 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311444 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:50.324096 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311447 4385 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:50.324096 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311449 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:50.324096 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311452 4385 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:50.324096 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311457 4385 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:50.324096 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311461 4385 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:50.324096 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311465 4385 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:50.324096 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311468 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:50.324096 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311470 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:50.324096 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311478 4385 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:50.324096 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311481 4385 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:50.324096 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311485 4385 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:50.324096 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311488 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:50.324691 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311491 4385 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:50.324691 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311494 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:50.324691 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311497 4385 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:50.324691 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311499 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:50.324691 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311502 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:50.324691 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311505 4385 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:50.324691 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311508 4385 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:50.324691 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311511 4385 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:50.324691 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311514 4385 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:50.324691 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311516 4385 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:50.324691 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311519 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:50.324691 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311522 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:50.324691 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311525 4385 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:50.324691 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311527 4385 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:50.324691 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311530 4385 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:50.324691 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311533 4385 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:50.324691 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311536 4385 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:50.324691 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311539 4385 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:50.324691 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311541 4385 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:50.324691 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311544 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:50.325343 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311547 4385 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:50.325343 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311549 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:50.325343 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311552 4385 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:50.325343 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311555 4385 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:50.325343 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311557 4385 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:50.325343 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311560 4385 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:50.325343 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311563 4385 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:50.325343 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311566 4385 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:50.325343 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311568 4385 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:50.325343 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311576 4385 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:50.325343 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311579 4385 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:50.325343 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311582 4385 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:50.325343 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311584 4385 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:50.325343 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311587 4385 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:50.325343 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311589 4385 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:50.325343 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311592 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:50.325343 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311595 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:50.325343 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311598 4385 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:50.325343 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311600 4385 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:50.325343 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311603 4385 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:50.325907 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.311629 4385 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 06:09:50.325907 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311819 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:50.325907 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311823 4385 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:50.325907 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311826 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:50.325907 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311843 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:50.325907 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311846 4385 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:50.325907 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311849 4385 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:50.325907 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311852 4385 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:50.325907 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311855 4385 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:50.325907 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311858 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:50.325907 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311860 4385 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:50.325907 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311863 4385 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:50.325907 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311866 4385 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:50.325907 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311868 4385 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:50.325907 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311871 4385 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:50.325907 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311874 4385 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:50.326316 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311877 4385 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:50.326316 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311879 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:50.326316 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311882 4385 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:50.326316 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311884 4385 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:50.326316 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311887 4385 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:50.326316 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311889 4385 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:50.326316 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311899 4385 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:50.326316 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311902 4385 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:50.326316 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311905 4385 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:50.326316 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311907 4385 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:50.326316 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311910 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:50.326316 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311913 4385 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:50.326316 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311916 4385 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:50.326316 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311919 4385 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:50.326316 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311921 4385 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:50.326316 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311924 4385 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:50.326316 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311926 4385 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:50.326316 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311929 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:50.326316 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311931 4385 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:50.326316 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311934 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:50.326787 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311937 4385 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:50.326787 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311940 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:50.326787 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311943 4385 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:50.326787 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311946 4385 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:50.326787 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311948 4385 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:50.326787 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311951 4385 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:50.326787 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311953 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:50.326787 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311956 4385 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:50.326787 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311959 4385 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:50.326787 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311961 4385 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:50.326787 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311964 4385 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:50.326787 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311966 4385 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:50.326787 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311969 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:50.326787 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311971 4385 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:50.326787 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311974 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:50.326787 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311976 4385 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:50.326787 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311979 4385 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:50.326787 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311982 4385 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:50.326787 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311984 4385 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:50.326787 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311992 4385 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:50.327285 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311995 4385 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:50.327285 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.311998 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:50.327285 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312000 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:50.327285 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312003 4385 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:50.327285 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312006 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:50.327285 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312008 4385 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:50.327285 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312011 4385 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:50.327285 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312013 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:50.327285 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312016 4385 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:50.327285 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312019 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:50.327285 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312021 4385 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:50.327285 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312024 4385 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:50.327285 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312028 4385 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:50.327285 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312032 4385 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:50.327285 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312036 4385 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:50.327285 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312039 4385 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:50.327285 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312042 4385 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:50.327285 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312050 4385 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:50.327285 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312053 4385 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:50.327795 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312056 4385 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:50.327795 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312058 4385 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:50.327795 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312061 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:50.327795 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312063 4385 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:50.327795 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312066 4385 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:50.327795 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312069 4385 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:50.327795 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312072 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:50.327795 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312074 4385 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:50.327795 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312077 4385 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:50.327795 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312080 4385 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:50.327795 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312083 4385 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:50.327795 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312086 4385 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:50.327795 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312088 4385 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:50.327795 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312098 4385 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:50.327795 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312102 4385 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:50.327795 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312105 4385 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:50.327795 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312107 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:50.327795 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312110 4385 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:50.327795 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312113 4385 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:50.327795 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312115 4385 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:50.328311 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312118 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:50.328311 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312120 4385 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:50.328311 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312123 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:50.328311 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312126 4385 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:50.328311 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312129 4385 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:50.328311 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312131 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:50.328311 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312134 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:50.328311 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312137 4385 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:50.328311 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312139 4385 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:50.328311 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312142 4385 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:50.328311 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312144 4385 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:50.328311 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312147 4385 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:50.328311 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.312171 4385 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 06:09:50.328311 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312208 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:50.328311 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312212 4385 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:50.328311 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312215 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:50.328698 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312218 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:50.328698 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312221 4385 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:50.328698 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312223 4385 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:50.328698 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312226 4385 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:50.328698 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312229 4385 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:50.328698 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312231 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:50.328698 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312234 4385 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:50.328698 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312237 4385 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:50.328698 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312239 4385 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:50.328698 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312242 4385 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:50.328698 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312250 4385 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:50.328698 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312253 4385 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:50.328698 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312255 4385 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:50.328698 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312258 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:50.328698 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312260 4385 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:50.328698 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312263 4385 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:50.328698 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312266 4385 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:50.328698 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312268 4385 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:50.328698 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312271 4385 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:50.328698 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312274 4385 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:50.329239 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312277 4385 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:50.329239 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312279 4385 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:50.329239 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312282 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:50.329239 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312285 4385 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:50.329239 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312287 4385 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:50.329239 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312290 4385 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:50.329239 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312292 4385 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:50.329239 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312295 4385 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:50.329239 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312298 4385 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:50.329239 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312300 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:50.329239 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312302 4385 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:50.329239 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312305 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:50.329239 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312308 4385 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:50.329239 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312310 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:50.329239 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312313 4385 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:50.329239 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312316 4385 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:50.329239 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312318 4385 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:50.329239 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312321 4385 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:50.329239 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312324 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:50.329239 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312326 4385 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:50.329707 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312329 4385 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:50.329707 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312331 4385 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:50.329707 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312334 4385 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:50.329707 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312342 4385 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:50.329707 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312345 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:50.329707 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312347 4385 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:50.329707 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312350 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:50.329707 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312353 4385 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:50.329707 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312356 4385 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:50.329707 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312359 4385 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:50.329707 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312361 4385 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:50.329707 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312364 4385 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:50.329707 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312366 4385 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:50.329707 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312369 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:50.329707 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312371 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:50.329707 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312374 4385 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:50.329707 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312376 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:50.329707 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312379 4385 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:50.329707 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312381 4385 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:50.330205 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312384 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:50.330205 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312386 4385 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:50.330205 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312389 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:50.330205 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312392 4385 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:50.330205 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312394 4385 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:50.330205 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312398 4385 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:50.330205 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312402 4385 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:50.330205 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312405 4385 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:50.330205 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312408 4385 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:50.330205 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312411 4385 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:50.330205 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312414 4385 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:50.330205 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312416 4385 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:50.330205 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312419 4385 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:50.330205 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312421 4385 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:50.330205 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312424 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:50.330205 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312427 4385 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:50.330205 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312429 4385 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:50.330205 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312438 4385 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:50.330205 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312441 4385 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:50.331009 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312443 4385 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:50.331009 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312446 4385 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:50.331009 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312449 4385 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:50.331009 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312451 4385 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:50.331009 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312454 4385 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:50.331009 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312457 4385 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:50.331009 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312461 4385 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:50.331009 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312464 4385 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:50.331009 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312468 4385 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:50.331009 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312471 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:50.331009 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312474 4385 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:50.331009 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312476 4385 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:50.331009 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312479 4385 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:50.331009 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312482 4385 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:50.331009 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312485 4385 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:50.331009 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312488 4385 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:50.331009 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312491 4385 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:50.331009 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312494 4385 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:50.331009 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312496 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:50.331009 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312499 4385 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:50.331785 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312501 4385 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:50.331785 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312504 4385 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:50.331785 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312507 4385 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:50.331785 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312510 4385 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:50.331785 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:50.312515 4385 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:50.331785 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.312577 4385 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 06:09:50.331785 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.313617 4385 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 06:09:50.331785 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.313710 4385 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 06:09:50.331785 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.313953 4385 server.go:1014] "Starting client certificate rotation" Nov 01 06:09:50.331785 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.314087 4385 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 06:09:50.331785 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.314140 4385 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 06:04:10 +0000 UTC" deadline="2028-06-26 21:34:48.899571858 +0000 UTC" Nov 01 06:09:50.331785 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.314167 4385 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14487h24m58.585406423s" Nov 01 06:09:50.331785 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.314494 4385 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 06:09:50.331785 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.317194 4385 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 06:09:50.331785 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.319326 4385 log.go:25] "Validated CRI v1 runtime API" Nov 01 06:09:50.331785 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.320918 4385 log.go:25] "Validated CRI v1 image API" Nov 01 06:09:50.332471 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.321703 4385 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 06:09:50.332471 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.323289 4385 fs.go:135] Filesystem UUIDs: map[1fb5b39b-261d-4c25-b15e-9a12c21bee9f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f19ef59e-80e2-4476-a60f-7c1f7b777a0d:/dev/nvme0n1p4] Nov 01 06:09:50.332552 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.323311 4385 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0158f578705e45b0aaf2d3520695748637340cc7d13adc72704385867a6eaeed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0158f578705e45b0aaf2d3520695748637340cc7d13adc72704385867a6eaeed/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/380de1f4b09bb1aa0a4e1a4e6fc44c27dc61d8eb48f5afe09509e396c0859d5f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/380de1f4b09bb1aa0a4e1a4e6fc44c27dc61d8eb48f5afe09509e396c0859d5f/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40d91ca4fd54e15d28ffc3f97d8aefa8cbe67c9028c9e3e5c1a7110fc9c84b38/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40d91ca4fd54e15d28ffc3f97d8aefa8cbe67c9028c9e3e5c1a7110fc9c84b38/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/43371c2322f44a7a149eabf907b11ee71d156e9808b02b50c71fdbb4a6f56625/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/43371c2322f44a7a149eabf907b11ee71d156e9808b02b50c71fdbb4a6f56625/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a512bd4e2f00eae2e9fb0e81325a8f953c9b5d0f8c7a60f7667351a3fcb3579/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a512bd4e2f00eae2e9fb0e81325a8f953c9b5d0f8c7a60f7667351a3fcb3579/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99ce3c1f6bf83c7eb09e0ae7bd283e9227842a2428109c76d9dbacfa335d43c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99ce3c1f6bf83c7eb09e0ae7bd283e9227842a2428109c76d9dbacfa335d43c8/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a337de93d5a379d6a2691299bc61da059212a2a24a8bdd8604c65e58b4e73735/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a337de93d5a379d6a2691299bc61da059212a2a24a8bdd8604c65e58b4e73735/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a35c2a7d5fb13e8b524daa7d29574a3cebc52e65475679c700dd5b307cfd3de5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a35c2a7d5fb13e8b524daa7d29574a3cebc52e65475679c700dd5b307cfd3de5/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5599496786fe856313aeae337a6feb17eae0eaf383dedf77fda113fae056a87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5599496786fe856313aeae337a6feb17eae0eaf383dedf77fda113fae056a87/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b213f89ca630f32c262ff8dca5cd1c7e79b4648ae2b632270c942b8e74d90ce3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b213f89ca630f32c262ff8dca5cd1c7e79b4648ae2b632270c942b8e74d90ce3/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0a7ae33424afe6f09493743c78d272fe9a55ac88e1c05518439767aefa7257d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0a7ae33424afe6f09493743c78d272fe9a55ac88e1c05518439767aefa7257d/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ddf6129b10d2dd9dc196acb809707ed5f454cb9b5d2e92f3e0cbc7854c0bd333/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ddf6129b10d2dd9dc196acb809707ed5f454cb9b5d2e92f3e0cbc7854c0bd333/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ead6d071a982365a9bbe51da31a7245fab8c10d2afe2cf92acda734f8015b671/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ead6d071a982365a9bbe51da31a7245fab8c10d2afe2cf92acda734f8015b671/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa0cb52b5dbb3cb0c44446945e595216808c2b27c8f78dd6a11506b57c09598a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa0cb52b5dbb3cb0c44446945e595216808c2b27c8f78dd6a11506b57c09598a/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/199a6bc2-3668-415c-8cc4-813f96eccfa4/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/199a6bc2-3668-415c-8cc4-813f96eccfa4/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271/volumes/kubernetes.io~projected/kube-api-access-4z2gw:{mountpoint:/var/lib/kubelet/pods/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271/volumes/kubernetes.io~projected/kube-api-access-4z2gw major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24ddad96-0a15-4c10-b302-416833e87746/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/24ddad96-0a15-4c10-b302-416833e87746/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24ddad96-0a15-4c10-b302-416833e87746/volumes/kubernetes.io~projected/kube-api-access-4vcgd:{mountpoint:/var/lib/kubelet/pods/24ddad96-0a15-4c10-b302-416833e87746/volumes/kubernetes.io~projected/kube-api-access-4vcgd major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24ddad96-0a15-4c10-b302-416833e87746/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/24ddad96-0a15-4c10-b302-416833e87746/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36effec9-edbb-40a4-8fe6-bee6b5bde3f1/volumes/kubernetes.io~projected/kube-api-access-n5pnm:{mountpoint:/var/lib/kubelet/pods/36effec9-edbb-40a4-8fe6-bee6b5bde3f1/volumes/kubernetes.io~projected/kube-api-access-n5pnm major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b623310-9483-47a2-a5de-f19b87819eda/volumes/kubernetes.io~projected/kube-api-access-rhqdn:{mountpoint:/var/lib/kubelet/pods/3b623310-9483-47a2-a5de-f19b87819eda/volumes/kubernetes.io~projected/kube-api-access-rhqdn major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45331df5-857e-4663-9131-6575bcd3095a/volumes/kubernetes.io~projected/kube-api-access-5xd5q:{mountpoint:/var/lib/kubelet/pods/45331df5-857e-4663-9131-6575bcd3095a/volumes/kubernetes.io~projected/kube-api-access-5xd5q major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45331df5-857e-4663-9131-6575bcd3095a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/45331df5-857e-4663-9131-6575bcd3095a/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/568704c3-e334-44e1-8373-12a873c305b5/volumes/kubernetes.io~projected/kube-api-access-6mqsk:{mountpoint:/var/lib/kubelet/pods/568704c3-e334-44e1-8373-12a873c305b5/volumes/kubernetes.io~projected/kube-api-access-6mqsk major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b0855c2-93a3-44a5-9311-63d891705e0e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/5b0855c2-93a3-44a5-9311-63d891705e0e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6228c388-659f-4ca7-8d51-4fa5658424e2/volumes/kubernetes.io~projected/kube-api-access-6wqjh:{mountpoint:/var/lib/kubelet/pods/6228c388-659f-4ca7-8d51-4fa5658424e2/volumes/kubernetes.io~projected/kube-api-access-6wqjh major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b856c58-9b9e-47b3-947a-ed56ae892e9b/volumes/kubernetes.io~projected/kube-api-access-njlqm:{mountpoint:/var/lib/kubelet/pods/7b856c58-9b9e-47b3-947a-ed56ae892e9b/volumes/kubernetes.io~projected/kube-api-access-njlqm major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8/volumes/kubernetes.io~projected/kube-api-access-qt9qk:{mountpoint:/var/lib/kubelet/pods/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8/volumes/kubernetes.io~projected/kube-api-access-qt9qk major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/939039b6-e071-4a27-b24e-20bb2669dc15/volumes/kubernetes.io~projected/kube-api-access-vd7r9:{mountpoint:/var/lib/kubelet/pods/939039b6-e071-4a27-b24e-20bb2669dc15/volumes/kubernetes.io~projected/kube-api-access-vd7r9 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93f3367a-3295-4f1c-931a-ba9c643d3b76/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/93f3367a-3295-4f1c-931a-ba9c643d3b76/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93f3367a-3295-4f1c-931a-ba9c643d3b76/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/93f3367a-3295-4f1c-931a-ba9c643d3b76/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93f3367a-3295-4f1c-931a-ba9c643d3b76/volumes/kubernetes.io~projected/kube-api-access-xr4ql:{mountpoint:/var/lib/kubelet/pods/93f3367a-3295-4f1c-931a-ba9c643d3b76/volumes/kubernetes.io~projected/kube-api-access-xr4ql major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b490df30-7bfc-4c25-a9f5-768733b3b6e6/volumes/kubernetes.io~projected/kube-api-access-rb7kp:{mountpoint:/var/lib/kubelet/pods/b490df30-7bfc-4c25-a9f5-768733b3b6e6/volumes/kubernetes.io~projected/kube-api-access-rb7kp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d48e417b-1b11-4ee3-8b45-0a71f68459bb/volumes/kubernetes.io~projected/kube-api-access-4tjfc:{mountpoint:/var/lib/kubelet/pods/d48e417b-1b11-4ee3-8b45-0a71f68459bb/volumes/kubernetes.io~projected/kube-api-access-4tjfc major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff209546-604b-462a-8f67-dee8fc8698d3/volumes/kubernetes.io~projected/kube-api-access-57g2w:{mountpoint:/var/lib/kubelet/pods/ff209546-604b-462a-8f67-dee8fc8698d3/volumes/kubernetes.io~projected/kube-api-access-57g2w major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff209546-604b-462a-8f67-dee8fc8698d3/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/ff209546-604b-462a-8f67-dee8fc8698d3/volumes/kubernetes.io~secret/signing-key major:0 minor:207 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6f3affa4ae45d21a50d45145332c71c1047c26c3e0d0730e66a77e8e069ee2f0/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/8c71ee31ee54e45a8332046815ec42e5462898da7221807196c3b70180b562e4/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/17a562ed01b049b937bbc21261596ae30b610c650738c6013fdbb5db6843515f/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/ce9416c97a8c46b823f94d9863945ff3562346b200aaded2d889b32a91842c55/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/fe39f2f50a3b47a592158b3e5f1e3d55e0dd3b79d1c2acb9af39f70d19be1a50/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/87ad797cf27a392102066527a61520f3e1af08136a75a61f61cf835c18c28794/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/77ec96e0af96da665453fb2967002110dd36dc41a1761ab0063dd7bb8140b89c/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d6471e17c75e081fefd551eb3a5cc3de58cf80cacc7f6b19a78caa7c35e12f30/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/cb247c86229432aef27cafd6bf977f5518519503c8230ce72ee1750be0fab508/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/5547e060eac0b59c5248f14214c06992c95f7c51bd8684d6f21b73a311db0ff1/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/b14340e3a9a468e4fe3238f8f8862bf6bede35c93760b995ef90bf291b410105/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/adf73d6c22fd065712fbed81c99eee4ff8d9f37bcbf06b2615f1cb350ee10ce6/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/93ad97bf443db6a7e5b5ad89844717d26404ce792847384de4074bdfd9a0712d/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-155:{mountpoint:/var/lib/containers/storage/overlay/a69717c0b046d16bc01c8bfb9d722292f7941477c30116366c3789198352de54/merged major:0 minor:155 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/78abac29f3c6bac67cab2158fc76ffd80ecb58d2ce11ea1439be02296deb3872/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/a70fb8a35c2a24c19b268d17a9bbdb1d73862617e2900958e4baf2985da76b33/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/7e42b42e60279ea7a2e99272da33680862ec9029428caad6771b2e0fb6efa954/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/be34a5f5139d54217554d148d9fd9350ebe3adff3314c245f1c87e07c1b10e7a/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c38ce829faad55e04ce3ca5ea1c14f28892a63a1960ee3f2ea67ee63cc68d3bd/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/7c9f450fe07bdec10aa8a006f6da5023763520ea35bc030b033864269db9f865/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/14147cc9fc87b38b6e768f19520f05147428fd14f17c06297b6247ac1ca53c44/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/f80351545fe4e39eff5c9af9bb7e66bd0eb08525482bf61d4d0b739e236092bf/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/adefcae38e98eccbc76c9ab2ca9336f3d7452799c85fd4bb888745630c1ec0c4/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/fa10fc4063375395d2bc2c7944ce5e2e5d9faf693b1af051ecf0af55b2ac1ed6/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/68e579507460c9d6e8ae0462af46fd51d786e688a420186ce6e336fcfd77a722/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/12f13dfe9dec49626292f9e4cfd530288b93652cb1ac2a6033bf99c639676efe/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/fb11b3e62f45d6beea06b0146bbe5e447ce67fd8d3e42f249bd3307da21332a3/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/d863772bed7d9c3fa14802bab780c09cc9c771bad7f9d22e2ca162e9bdd8f99a/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/586e497554d155fba9868e13ce19b2dbab489c672ea4c030618471553a299c2b/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/8b3ab326f913f1f62b17b15bd38d4a8ae409796897cd48d2cf4911c120b6483b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/68e212d7d6184b3206c74c63403e6a7e32bb2fe73d381af65b9fc2657c8bbbf9/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/843aba5366b1fe72fc41d269b1452f83982d6d496acf3f0fc39830ef8ca67f79/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/c44d18458a9e487141b5d339b929eb8d2e25e16b3800414afa2acf30febe4e48/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/287ac8919f9903066f8e803cdb0f11c2c4546bb0528bf9ecaf2fbd8eb18c9727/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/8c28ce215f705a2a3e2403224d268b586e09c458272ce7ad43d3a708b435e57a/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/c89803d85be7db73939d728d2fa50ea74822db340ba303918493b26c4421dde1/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/6d018e5b19c8f551ddf37eef2209a26e43a567673f4f5a7670f4e87af313b583/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/fb51f2e169bff6475b097854fc1f2fc1e1cab04de0b46e7658316c86e125dc83/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/3bab58ea29b391c8e58957dbf7208486e1b9c3670f0ba3cf2271e72e337b3aec/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 01 06:09:50.332715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.329816 4385 manager.go:217] Machine: {Timestamp:2026-11-01 06:09:50.329357654 +0000 UTC m=+0.093157502 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097868 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c53dd4d080a6e06eb0162035b0773 SystemUUID:ec2c53dd-4d08-0a6e-06eb-0162035b0773 BootID:6cc7d41c-5410-4001-8366-8ead92311cc5 Filesystems:[{Device:/var/lib/kubelet/pods/45331df5-857e-4663-9131-6575bcd3095a/volumes/kubernetes.io~projected/kube-api-access-5xd5q DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/93f3367a-3295-4f1c-931a-ba9c643d3b76/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/93f3367a-3295-4f1c-931a-ba9c643d3b76/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5599496786fe856313aeae337a6feb17eae0eaf383dedf77fda113fae056a87/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36effec9-edbb-40a4-8fe6-bee6b5bde3f1/volumes/kubernetes.io~projected/kube-api-access-n5pnm DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/939039b6-e071-4a27-b24e-20bb2669dc15/volumes/kubernetes.io~projected/kube-api-access-vd7r9 DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b213f89ca630f32c262ff8dca5cd1c7e79b4648ae2b632270c942b8e74d90ce3/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40d91ca4fd54e15d28ffc3f97d8aefa8cbe67c9028c9e3e5c1a7110fc9c84b38/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ddf6129b10d2dd9dc196acb809707ed5f454cb9b5d2e92f3e0cbc7854c0bd333/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/199a6bc2-3668-415c-8cc4-813f96eccfa4/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24ddad96-0a15-4c10-b302-416833e87746/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3b623310-9483-47a2-a5de-f19b87819eda/volumes/kubernetes.io~projected/kube-api-access-rhqdn DeviceMajor:0 DeviceMinor:54 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b490df30-7bfc-4c25-a9f5-768733b3b6e6/volumes/kubernetes.io~projected/kube-api-access-rb7kp DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a512bd4e2f00eae2e9fb0e81325a8f953c9b5d0f8c7a60f7667351a3fcb3579/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a337de93d5a379d6a2691299bc61da059212a2a24a8bdd8604c65e58b4e73735/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d48e417b-1b11-4ee3-8b45-0a71f68459bb/volumes/kubernetes.io~projected/kube-api-access-4tjfc DeviceMajor:0 DeviceMinor:51 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8/volumes/kubernetes.io~projected/kube-api-access-qt9qk DeviceMajor:0 DeviceMinor:53 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ff209546-604b-462a-8f67-dee8fc8698d3/volumes/kubernetes.io~projected/kube-api-access-57g2w DeviceMajor:0 DeviceMinor:208 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/7b856c58-9b9e-47b3-947a-ed56ae892e9b/volumes/kubernetes.io~projected/kube-api-access-njlqm DeviceMajor:0 DeviceMinor:52 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24ddad96-0a15-4c10-b302-416833e87746/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/5b0855c2-93a3-44a5-9311-63d891705e0e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:213 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99ce3c1f6bf83c7eb09e0ae7bd283e9227842a2428109c76d9dbacfa335d43c8/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a35c2a7d5fb13e8b524daa7d29574a3cebc52e65475679c700dd5b307cfd3de5/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-155 DeviceMajor:0 DeviceMinor:155 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271/volumes/kubernetes.io~projected/kube-api-access-4z2gw DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/43371c2322f44a7a149eabf907b11ee71d156e9808b02b50c71fdbb4a6f56625/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6228c388-659f-4ca7-8d51-4fa5658424e2/volumes/kubernetes.io~projected/kube-api-access-6wqjh DeviceMajor:0 DeviceMinor:214 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0158f578705e45b0aaf2d3520695748637340cc7d13adc72704385867a6eaeed/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ff209546-604b-462a-8f67-dee8fc8698d3/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:207 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ead6d071a982365a9bbe51da31a7245fab8c10d2afe2cf92acda734f8015b671/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0a7ae33424afe6f09493743c78d272fe9a55ac88e1c05518439767aefa7257d/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/380de1f4b09bb1aa0a4e1a4e6fc44c27dc61d8eb48f5afe09509e396c0859d5f/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45331df5-857e-4663-9131-6575bcd3095a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/568704c3-e334-44e1-8373-12a873c305b5/volumes/kubernetes.io~projected/kube-api-access-6mqsk DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa0cb52b5dbb3cb0c44446945e595216808c2b27c8f78dd6a11506b57c09598a/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/93f3367a-3295-4f1c-931a-ba9c643d3b76/volumes/kubernetes.io~projected/kube-api-access-xr4ql DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/24ddad96-0a15-4c10-b302-416833e87746/volumes/kubernetes.io~projected/kube-api-access-4vcgd DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0158f578705e45b MacAddress:42:14:96:ae:2f:8f Speed:10000 Mtu:8901} {Name:99ce3c1f6bf83c7 MacAddress:96:7f:cf:c8:25:97 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:52:0a:10:b7:43 Speed:0 Mtu:9001} {Name:br-int MacAddress:66:07:d0:a5:f8:2a Speed:0 Mtu:8901} {Name:ddf6129b10d2dd9 MacAddress:e6:71:35:30:a7:d0 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:52:0a:10:b7:43 Speed:0 Mtu:9001} {Name:fa0cb52b5dbb3cb MacAddress:42:c5:c7:bd:73:fc Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:22:33:ec:48:2c:ae Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:46:c8:8a:2e:4a:c6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 06:09:50.332715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.330092 4385 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 06:09:50.332715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.330139 4385 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 06:09:50.332715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.330510 4385 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 06:09:50.332715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.330535 4385 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-68.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 06:09:50.332715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.330733 4385 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 06:09:50.332715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.330745 4385 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 06:09:50.332715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.330763 4385 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 06:09:50.332715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.330790 4385 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 06:09:50.332715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.330930 4385 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 06:09:50.332715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.331165 4385 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 06:09:50.332715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.331310 4385 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 06:09:50.332715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.331357 4385 kubelet.go:493] "Attempting to sync node with API server" Nov 01 06:09:50.332715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.331370 4385 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 06:09:50.332715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.331384 4385 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 06:09:50.332715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.331399 4385 kubelet.go:405] "Adding apiserver pod source" Nov 01 06:09:50.332715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.331411 4385 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 06:09:50.332715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.332190 4385 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 06:09:50.332715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.332206 4385 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 06:09:50.333301 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.332749 4385 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 06:09:50.335902 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.335880 4385 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 06:09:50.335902 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.335902 4385 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 06:09:50.336110 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.336097 4385 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 06:09:50.336639 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.336621 4385 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 06:09:50.336693 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.336655 4385 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 06:09:50.336773 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.336764 4385 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 06:09:50.336805 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.336779 4385 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 06:09:50.336805 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.336786 4385 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 06:09:50.336805 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.336792 4385 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 06:09:50.336805 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.336798 4385 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 06:09:50.336805 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.336803 4385 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 06:09:50.337030 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.336809 4385 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 06:09:50.337030 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.336815 4385 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 06:09:50.337030 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.336822 4385 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 06:09:50.337030 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.336827 4385 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 06:09:50.337030 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.336927 4385 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 06:09:50.337030 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.336936 4385 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 06:09:50.337030 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.336963 4385 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 06:09:50.337030 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.336969 4385 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 06:09:50.340396 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.340383 4385 server.go:1263] "Started kubelet" Nov 01 06:09:50.340481 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.340453 4385 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 06:09:50.340950 ip-10-0-140-68 systemd[1]: Started Kubernetes Kubelet. Nov 01 06:09:50.341556 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.340869 4385 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 06:09:50.341556 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.341029 4385 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 06:09:50.341739 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.341721 4385 server.go:317] "Adding debug handlers to kubelet server" Nov 01 06:09:50.341892 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.341873 4385 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.343074 4385 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.343133 4385 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.343235 4385 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.343244 4385 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:50.343285 4385 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-68.us-west-1.compute.internal\" not found" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.343464 4385 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.344865 4385 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 06:04:11 +0000 UTC" deadline="2028-05-06 21:58:11.609501336 +0000 UTC" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345161 4385 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13263h48m21.264345846s" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345321 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24ddad96-0a15-4c10-b302-416833e87746" volumeName="kubernetes.io/configmap/24ddad96-0a15-4c10-b302-416833e87746-ovnkube-config" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345355 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="568704c3-e334-44e1-8373-12a873c305b5" volumeName="kubernetes.io/projected/568704c3-e334-44e1-8373-12a873c305b5-kube-api-access-6mqsk" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345371 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b856c58-9b9e-47b3-947a-ed56ae892e9b" volumeName="kubernetes.io/configmap/7b856c58-9b9e-47b3-947a-ed56ae892e9b-serviceca" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345386 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff209546-604b-462a-8f67-dee8fc8698d3" volumeName="kubernetes.io/projected/ff209546-604b-462a-8f67-dee8fc8698d3-kube-api-access-57g2w" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345401 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24ddad96-0a15-4c10-b302-416833e87746" volumeName="kubernetes.io/projected/24ddad96-0a15-4c10-b302-416833e87746-kube-api-access-4vcgd" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345413 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24ddad96-0a15-4c10-b302-416833e87746" volumeName="kubernetes.io/secret/24ddad96-0a15-4c10-b302-416833e87746-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345428 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b623310-9483-47a2-a5de-f19b87819eda" volumeName="kubernetes.io/configmap/3b623310-9483-47a2-a5de-f19b87819eda-cni-binary-copy" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345442 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="568704c3-e334-44e1-8373-12a873c305b5" volumeName="kubernetes.io/configmap/568704c3-e334-44e1-8373-12a873c305b5-config-volume" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345459 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d48e417b-1b11-4ee3-8b45-0a71f68459bb" volumeName="kubernetes.io/configmap/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-daemon-config" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345474 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45331df5-857e-4663-9131-6575bcd3095a" volumeName="kubernetes.io/secret/45331df5-857e-4663-9131-6575bcd3095a-cert" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345486 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6228c388-659f-4ca7-8d51-4fa5658424e2" volumeName="kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345499 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93f3367a-3295-4f1c-931a-ba9c643d3b76" volumeName="kubernetes.io/empty-dir/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-tuned" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345513 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93f3367a-3295-4f1c-931a-ba9c643d3b76" volumeName="kubernetes.io/projected/93f3367a-3295-4f1c-931a-ba9c643d3b76-kube-api-access-xr4ql" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345527 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff209546-604b-462a-8f67-dee8fc8698d3" volumeName="kubernetes.io/secret/ff209546-604b-462a-8f67-dee8fc8698d3-signing-key" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345543 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8" volumeName="kubernetes.io/projected/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-kube-api-access-qt9qk" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345557 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="939039b6-e071-4a27-b24e-20bb2669dc15" volumeName="kubernetes.io/empty-dir/939039b6-e071-4a27-b24e-20bb2669dc15-tmp-dir" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345568 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="939039b6-e071-4a27-b24e-20bb2669dc15" volumeName="kubernetes.io/projected/939039b6-e071-4a27-b24e-20bb2669dc15-kube-api-access-vd7r9" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345581 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="199a6bc2-3668-415c-8cc4-813f96eccfa4" volumeName="kubernetes.io/configmap/199a6bc2-3668-415c-8cc4-813f96eccfa4-konnectivity-ca" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345596 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36effec9-edbb-40a4-8fe6-bee6b5bde3f1" volumeName="kubernetes.io/empty-dir/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-data-volume" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345610 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b856c58-9b9e-47b3-947a-ed56ae892e9b" volumeName="kubernetes.io/projected/7b856c58-9b9e-47b3-947a-ed56ae892e9b-kube-api-access-njlqm" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345624 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b490df30-7bfc-4c25-a9f5-768733b3b6e6" volumeName="kubernetes.io/configmap/b490df30-7bfc-4c25-a9f5-768733b3b6e6-iptables-alerter-script" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345636 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d48e417b-1b11-4ee3-8b45-0a71f68459bb" volumeName="kubernetes.io/projected/d48e417b-1b11-4ee3-8b45-0a71f68459bb-kube-api-access-4tjfc" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345646 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff209546-604b-462a-8f67-dee8fc8698d3" volumeName="kubernetes.io/configmap/ff209546-604b-462a-8f67-dee8fc8698d3-signing-cabundle" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345658 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="199a6bc2-3668-415c-8cc4-813f96eccfa4" volumeName="kubernetes.io/secret/199a6bc2-3668-415c-8cc4-813f96eccfa4-agent-certs" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345670 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24ddad96-0a15-4c10-b302-416833e87746" volumeName="kubernetes.io/configmap/24ddad96-0a15-4c10-b302-416833e87746-ovnkube-script-lib" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345685 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24ddad96-0a15-4c10-b302-416833e87746" volumeName="kubernetes.io/configmap/24ddad96-0a15-4c10-b302-416833e87746-env-overrides" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345698 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36effec9-edbb-40a4-8fe6-bee6b5bde3f1" volumeName="kubernetes.io/configmap/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345712 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b623310-9483-47a2-a5de-f19b87819eda" volumeName="kubernetes.io/configmap/3b623310-9483-47a2-a5de-f19b87819eda-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345740 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b623310-9483-47a2-a5de-f19b87819eda" volumeName="kubernetes.io/projected/3b623310-9483-47a2-a5de-f19b87819eda-kube-api-access-rhqdn" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345752 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b0855c2-93a3-44a5-9311-63d891705e0e" volumeName="kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345766 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b490df30-7bfc-4c25-a9f5-768733b3b6e6" volumeName="kubernetes.io/projected/b490df30-7bfc-4c25-a9f5-768733b3b6e6-kube-api-access-rb7kp" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345777 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d48e417b-1b11-4ee3-8b45-0a71f68459bb" volumeName="kubernetes.io/configmap/d48e417b-1b11-4ee3-8b45-0a71f68459bb-cni-binary-copy" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345789 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b623310-9483-47a2-a5de-f19b87819eda" volumeName="kubernetes.io/configmap/3b623310-9483-47a2-a5de-f19b87819eda-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345800 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36effec9-edbb-40a4-8fe6-bee6b5bde3f1" volumeName="kubernetes.io/projected/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-api-access-n5pnm" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345811 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45331df5-857e-4663-9131-6575bcd3095a" volumeName="kubernetes.io/projected/45331df5-857e-4663-9131-6575bcd3095a-kube-api-access-5xd5q" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345847 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="568704c3-e334-44e1-8373-12a873c305b5" volumeName="kubernetes.io/empty-dir/568704c3-e334-44e1-8373-12a873c305b5-tmp-dir" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345860 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93f3367a-3295-4f1c-931a-ba9c643d3b76" volumeName="kubernetes.io/empty-dir/93f3367a-3295-4f1c-931a-ba9c643d3b76-tmp" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345871 4385 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e17cd4c-13b9-4dbf-8798-5fc68eaa1271" volumeName="kubernetes.io/projected/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-kube-api-access-4z2gw" seLinuxMountContext="" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345881 4385 reconstruct.go:97] "Volume reconstruction finished" Nov 01 06:09:50.346015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.345888 4385 reconciler.go:29] "Reconciler: start to sync state" Nov 01 06:09:50.348332 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.346776 4385 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 06:09:50.348332 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.346787 4385 factory.go:55] Registering systemd factory Nov 01 06:09:50.348332 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.346795 4385 factory.go:223] Registration of the systemd container factory successfully Nov 01 06:09:50.348332 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.347060 4385 factory.go:153] Registering CRI-O factory Nov 01 06:09:50.348332 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.347086 4385 factory.go:223] Registration of the crio container factory successfully Nov 01 06:09:50.348332 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.347111 4385 factory.go:103] Registering Raw factory Nov 01 06:09:50.348332 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.347133 4385 manager.go:1199] Started watching for new ooms in manager Nov 01 06:09:50.348332 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.347763 4385 manager.go:319] Starting recovery of all containers Nov 01 06:09:50.362987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.362935 4385 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 06:09:50.376515 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.376480 4385 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 06:09:50.376515 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.376513 4385 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 06:09:50.376711 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.376536 4385 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 06:09:50.376711 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:50.376575 4385 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 06:09:50.391351 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.391315 4385 generic.go:356] "Generic (PLEG): container finished" podID="58599ac63e6a6b0b2eb7aede2e6e233a" containerID="c7a26ddd76fe46720d9c0356a72dc73bc750c01e016da41f2fff42257c167da6" exitCode=0 Nov 01 06:09:50.412082 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.412039 4385 generic.go:356] "Generic (PLEG): container finished" podID="3b623310-9483-47a2-a5de-f19b87819eda" containerID="4e633c9b8d58ddfe900be37963120338e74861b5c539c76b15b21c45c67ed06d" exitCode=0 Nov 01 06:09:50.412082 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.412074 4385 generic.go:356] "Generic (PLEG): container finished" podID="3b623310-9483-47a2-a5de-f19b87819eda" containerID="90e150b3ba937632ac7584524dbac6844faf1fd36a383ee2c397e66a01a4b403" exitCode=0 Nov 01 06:09:50.412082 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.412084 4385 generic.go:356] "Generic (PLEG): container finished" podID="3b623310-9483-47a2-a5de-f19b87819eda" containerID="7f03e1809f418a0a1da6fdf456a8ad0a45a4207b287fa37a41e858932aa15e4c" exitCode=0 Nov 01 06:09:50.412359 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.412093 4385 generic.go:356] "Generic (PLEG): container finished" podID="3b623310-9483-47a2-a5de-f19b87819eda" containerID="dd7533cbebfea44f33a150517560396b834415ed3d606dc0e2172becd127bcca" exitCode=0 Nov 01 06:09:50.412359 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.412101 4385 generic.go:356] "Generic (PLEG): container finished" podID="3b623310-9483-47a2-a5de-f19b87819eda" containerID="e900319af2df00f1628e273931cd34643fb472938c77bb8bd34e830259dfd95e" exitCode=0 Nov 01 06:09:50.412359 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.412109 4385 generic.go:356] "Generic (PLEG): container finished" podID="3b623310-9483-47a2-a5de-f19b87819eda" containerID="275a4717cc5ae5dfc425b7c32987dfbb88c1b0fdc174ae4817005fe76ebbe4d5" exitCode=0 Nov 01 06:09:50.415016 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.414995 4385 manager.go:324] Recovery completed Nov 01 06:09:50.427937 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.427915 4385 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:50.429325 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.429306 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:50.429426 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.429344 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:50.429426 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.429361 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:50.430797 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.430779 4385 cpu_manager.go:226] "Starting" policy="none" Nov 01 06:09:50.430797 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.430792 4385 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 06:09:50.430952 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.430810 4385 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 06:09:50.430952 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.430793 4385 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:50.430952 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.430947 4385 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 06:09:50.431037 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.430958 4385 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 06:09:50.431037 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.430978 4385 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 06:09:50.431037 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.430986 4385 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 06:09:50.431037 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.430992 4385 policy_none.go:50] "Start" Nov 01 06:09:50.431037 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.431000 4385 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 06:09:50.431037 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.431009 4385 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 06:09:50.431222 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.431142 4385 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 06:09:50.431222 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.431152 4385 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 06:09:50.431222 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.431159 4385 policy_none.go:44] "Start" Nov 01 06:09:50.431868 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.431851 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:50.431924 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.431886 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:50.431924 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.431902 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:50.435335 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.435278 4385 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 06:09:50.438278 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.438261 4385 manager.go:341] "Starting Device Plugin manager" Nov 01 06:09:50.438363 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:50.438288 4385 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 06:09:50.438363 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.438298 4385 server.go:87] "Starting device plugin registration server" Nov 01 06:09:50.438894 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.438820 4385 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 06:09:50.438894 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.438860 4385 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 06:09:50.439257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.439235 4385 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 06:09:50.439605 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.439556 4385 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 06:09:50.439684 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.439620 4385 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 06:09:50.439684 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.439630 4385 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 06:09:50.439852 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.439719 4385 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T06:09:50.439604797Z","UUID":"ef91ec3c-04ea-4551-8360-1ea87c9103dd","Handler":null,"Name":"","Endpoint":""} Nov 01 06:09:50.441904 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.441885 4385 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 06:09:50.442023 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.441945 4385 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 06:09:50.442565 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:50.442543 4385 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 06:09:50.442646 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:50.442611 4385 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-68.us-west-1.compute.internal\" not found" Nov 01 06:09:50.477957 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.477422 4385 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal"] Nov 01 06:09:50.478112 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.478063 4385 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:50.479315 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.479297 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:50.479315 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.479327 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:50.479479 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.479344 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:50.479613 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.479602 4385 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:50.479770 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.479753 4385 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:50.480607 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.480585 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:50.480607 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.480594 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:50.480725 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.480622 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:50.480725 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.480636 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:50.480792 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.480623 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:50.480792 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.480784 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:50.480924 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.480895 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" event={"ID":"58599ac63e6a6b0b2eb7aede2e6e233a","Type":"ContainerStarted","Data":"a028da7f9fbd18a3456c0944599c78c30ac81f562993e184222f9cbbaf2c6129"} Nov 01 06:09:50.480978 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.480935 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" event={"ID":"58599ac63e6a6b0b2eb7aede2e6e233a","Type":"ContainerDied","Data":"c7a26ddd76fe46720d9c0356a72dc73bc750c01e016da41f2fff42257c167da6"} Nov 01 06:09:50.480978 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.480946 4385 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:50.480978 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.480955 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" event={"ID":"58599ac63e6a6b0b2eb7aede2e6e233a","Type":"ContainerStarted","Data":"c0a7ae33424afe6f09493743c78d272fe9a55ac88e1c05518439767aefa7257d"} Nov 01 06:09:50.481121 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.480980 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal" event={"ID":"645c51de28f312aaa75d7c297b03488b","Type":"ContainerStarted","Data":"b52f53e5e2278966472f585f97b01eaed9dfa2a6d3c3081ddada2d96b55827fe"} Nov 01 06:09:50.481121 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.480998 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal" event={"ID":"645c51de28f312aaa75d7c297b03488b","Type":"ContainerStarted","Data":"40d91ca4fd54e15d28ffc3f97d8aefa8cbe67c9028c9e3e5c1a7110fc9c84b38"} Nov 01 06:09:50.481121 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:50.481097 4385 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-68.us-west-1.compute.internal\" not found" node="ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:50.481856 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.481819 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:50.481922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.481865 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:50.481922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.481882 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:50.482032 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:50.482020 4385 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-68.us-west-1.compute.internal\" not found" node="ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:50.539088 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.539056 4385 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:50.539088 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.539078 4385 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 06:09:50.539287 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.539179 4385 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 06:09:50.539471 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.539457 4385 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 06:09:50.542008 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.541978 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:50.542131 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.542030 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:50.542131 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.542041 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:50.542131 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.542089 4385 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:50.589565 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.589532 4385 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 06:09:50.589814 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.589791 4385 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 06:09:50.639899 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.639864 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/645c51de28f312aaa75d7c297b03488b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal\" (UID: \"645c51de28f312aaa75d7c297b03488b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:50.639899 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.639899 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/58599ac63e6a6b0b2eb7aede2e6e233a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal\" (UID: \"58599ac63e6a6b0b2eb7aede2e6e233a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:50.640071 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.639920 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58599ac63e6a6b0b2eb7aede2e6e233a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal\" (UID: \"58599ac63e6a6b0b2eb7aede2e6e233a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:50.643973 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.643952 4385 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:50.689328 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.689251 4385 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:50.689328 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.689329 4385 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:50.689927 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.689910 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeNotReady" Nov 01 06:09:50.689991 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.689945 4385 setters.go:546] "Node became not ready" node="ip-10-0-140-68.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-01T06:09:50Z","lastTransitionTime":"2026-11-01T06:09:50Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 01 06:09:50.701443 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.701421 4385 nodeinfomanager.go:471] existing CSINode "ip-10-0-140-68.us-west-1.compute.internal" is owned by different node (oldNodeID="5aecce05-0755-49e6-abfd-54198f08def1", newNodeID=""), cleaning up... Nov 01 06:09:50.701944 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.701928 4385 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 06:09:50.702037 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:50.701968 4385 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:50.702037 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.701989 4385 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:50.740092 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.740055 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/58599ac63e6a6b0b2eb7aede2e6e233a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal\" (UID: \"58599ac63e6a6b0b2eb7aede2e6e233a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:50.740092 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.740091 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58599ac63e6a6b0b2eb7aede2e6e233a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal\" (UID: \"58599ac63e6a6b0b2eb7aede2e6e233a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:50.740306 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.740108 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/645c51de28f312aaa75d7c297b03488b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal\" (UID: \"645c51de28f312aaa75d7c297b03488b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:50.740306 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.740161 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/645c51de28f312aaa75d7c297b03488b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal\" (UID: \"645c51de28f312aaa75d7c297b03488b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:50.740306 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.740169 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/58599ac63e6a6b0b2eb7aede2e6e233a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal\" (UID: \"58599ac63e6a6b0b2eb7aede2e6e233a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:50.740306 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.740182 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58599ac63e6a6b0b2eb7aede2e6e233a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal\" (UID: \"58599ac63e6a6b0b2eb7aede2e6e233a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:50.759669 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:50.759646 4385 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 06:09:50.759825 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:50.759700 4385 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:51.331695 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.331654 4385 apiserver.go:52] "Watching apiserver" Nov 01 06:09:51.416563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.416537 4385 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:51.416699 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.416673 4385 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:51.434383 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.434350 4385 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 06:09:51.434820 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.434798 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb","openshift-ovn-kubernetes/ovnkube-node-wj2f6","kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal","openshift-image-registry/node-ca-vth7q","openshift-network-operator/iptables-alerter-rb97g","openshift-ingress-canary/ingress-canary-2vfjz","openshift-service-ca/service-ca-5fffc97bdf-ssrz5","openshift-dns/dns-default-62hgp","openshift-dns/node-resolver-ks27g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-258pf","openshift-multus/network-metrics-daemon-b98x2","openshift-network-diagnostics/network-check-target-vvpd2","openshift-insights/insights-runtime-extractor-b9rlh","openshift-multus/multus-sxts6","kube-system/global-pull-secret-syncer-txvqp","kube-system/konnectivity-agent-h8j7v","openshift-multus/multus-additional-cni-plugins-fvnw6"] Nov 01 06:09:51.440894 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.440870 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:51.442421 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.442402 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:51.442816 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.442798 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:51.444320 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.444302 4385 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 06:09:51.445680 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.445663 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568704c3-e334-44e1-8373-12a873c305b5-tmp-dir\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:51.445734 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.445692 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-registration-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:51.445734 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.445709 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-device-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:51.445734 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.445724 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xd5q\" (UniqueName: \"kubernetes.io/projected/45331df5-857e-4663-9131-6575bcd3095a-kube-api-access-5xd5q\") pod \"ingress-canary-2vfjz\" (UID: \"45331df5-857e-4663-9131-6575bcd3095a\") " pod="openshift-ingress-canary/ingress-canary-2vfjz" Nov 01 06:09:51.445885 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.445740 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-modprobe-d\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.445885 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.445792 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ff209546-604b-462a-8f67-dee8fc8698d3-signing-cabundle\") pod \"service-ca-5fffc97bdf-ssrz5\" (UID: \"ff209546-604b-462a-8f67-dee8fc8698d3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ssrz5" Nov 01 06:09:51.445885 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.445865 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-var-lib-kubelet\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.445885 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.445882 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z2gw\" (UniqueName: \"kubernetes.io/projected/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-kube-api-access-4z2gw\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:51.446048 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.445917 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568704c3-e334-44e1-8373-12a873c305b5-tmp-dir\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:51.446048 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.445991 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93f3367a-3295-4f1c-931a-ba9c643d3b76-tmp\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.446048 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446020 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b623310-9483-47a2-a5de-f19b87819eda-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:51.446132 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446073 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-socket-dir-parent\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.446132 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446082 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93f3367a-3295-4f1c-931a-ba9c643d3b76-tmp\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.446132 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446106 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:51.446221 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446133 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-run-systemd\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.446221 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446156 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-run-ovn\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.446221 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446182 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/24ddad96-0a15-4c10-b302-416833e87746-env-overrides\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.446221 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446211 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-sysctl-d\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.446348 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446228 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd7r9\" (UniqueName: \"kubernetes.io/projected/939039b6-e071-4a27-b24e-20bb2669dc15-kube-api-access-vd7r9\") pod \"node-resolver-ks27g\" (UID: \"939039b6-e071-4a27-b24e-20bb2669dc15\") " pod="openshift-dns/node-resolver-ks27g" Nov 01 06:09:51.446348 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446263 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-cni-dir\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.446348 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446288 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-run-k8s-cni-cncf-io\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.446348 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446313 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-var-lib-cni-bin\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.446662 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446356 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-kubelet\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.446662 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446400 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/24ddad96-0a15-4c10-b302-416833e87746-ovnkube-config\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.446662 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446439 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b856c58-9b9e-47b3-947a-ed56ae892e9b-serviceca\") pod \"node-ca-vth7q\" (UID: \"7b856c58-9b9e-47b3-947a-ed56ae892e9b\") " pod="openshift-image-registry/node-ca-vth7q" Nov 01 06:09:51.446662 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446465 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-kubernetes\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.446662 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446490 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-systemd\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.446662 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446511 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-host\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.446662 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446534 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b623310-9483-47a2-a5de-f19b87819eda-cnibin\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:51.446662 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446579 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-node-log\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.446662 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446604 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/568704c3-e334-44e1-8373-12a873c305b5-config-volume\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:51.446662 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446640 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:51.447084 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446690 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-sysconfig\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.447084 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446715 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b623310-9483-47a2-a5de-f19b87819eda-cni-binary-copy\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:51.447084 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446749 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vcgd\" (UniqueName: \"kubernetes.io/projected/24ddad96-0a15-4c10-b302-416833e87746-kube-api-access-4vcgd\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.447084 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446858 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb7kp\" (UniqueName: \"kubernetes.io/projected/b490df30-7bfc-4c25-a9f5-768733b3b6e6-kube-api-access-rb7kp\") pod \"iptables-alerter-rb97g\" (UID: \"b490df30-7bfc-4c25-a9f5-768733b3b6e6\") " pod="openshift-network-operator/iptables-alerter-rb97g" Nov 01 06:09:51.447084 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446904 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45331df5-857e-4663-9131-6575bcd3095a-cert\") pod \"ingress-canary-2vfjz\" (UID: \"45331df5-857e-4663-9131-6575bcd3095a\") " pod="openshift-ingress-canary/ingress-canary-2vfjz" Nov 01 06:09:51.447084 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446923 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njlqm\" (UniqueName: \"kubernetes.io/projected/7b856c58-9b9e-47b3-947a-ed56ae892e9b-kube-api-access-njlqm\") pod \"node-ca-vth7q\" (UID: \"7b856c58-9b9e-47b3-947a-ed56ae892e9b\") " pod="openshift-image-registry/node-ca-vth7q" Nov 01 06:09:51.447084 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446949 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt9qk\" (UniqueName: \"kubernetes.io/projected/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-kube-api-access-qt9qk\") pod \"network-metrics-daemon-b98x2\" (UID: \"8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8\") " pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:51.447084 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446966 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/939039b6-e071-4a27-b24e-20bb2669dc15-hosts-file\") pod \"node-resolver-ks27g\" (UID: \"939039b6-e071-4a27-b24e-20bb2669dc15\") " pod="openshift-dns/node-resolver-ks27g" Nov 01 06:09:51.447084 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.446990 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqdn\" (UniqueName: \"kubernetes.io/projected/3b623310-9483-47a2-a5de-f19b87819eda-kube-api-access-rhqdn\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:51.447454 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447095 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-cnibin\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.447454 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447127 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5pnm\" (UniqueName: \"kubernetes.io/projected/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-api-access-n5pnm\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:51.447454 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447172 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-cni-netd\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.447454 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447217 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/199a6bc2-3668-415c-8cc4-813f96eccfa4-konnectivity-ca\") pod \"konnectivity-agent-h8j7v\" (UID: \"199a6bc2-3668-415c-8cc4-813f96eccfa4\") " pod="kube-system/konnectivity-agent-h8j7v" Nov 01 06:09:51.447454 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447244 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-tuned\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.447454 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447272 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b623310-9483-47a2-a5de-f19b87819eda-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:51.447454 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447303 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-system-cni-dir\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.447454 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447358 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-tuned\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.447454 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447380 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:51.447454 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447409 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-log-socket\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.447454 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447427 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b0855c2-93a3-44a5-9311-63d891705e0e-dbus\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:51.447454 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447447 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/939039b6-e071-4a27-b24e-20bb2669dc15-tmp-dir\") pod \"node-resolver-ks27g\" (UID: \"939039b6-e071-4a27-b24e-20bb2669dc15\") " pod="openshift-dns/node-resolver-ks27g" Nov 01 06:09:51.447822 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447478 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-var-lib-openvswitch\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.447822 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447511 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-run-openvswitch\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.447822 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447521 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/939039b6-e071-4a27-b24e-20bb2669dc15-tmp-dir\") pod \"node-resolver-ks27g\" (UID: \"939039b6-e071-4a27-b24e-20bb2669dc15\") " pod="openshift-dns/node-resolver-ks27g" Nov 01 06:09:51.447822 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447534 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b490df30-7bfc-4c25-a9f5-768733b3b6e6-host-slash\") pod \"iptables-alerter-rb97g\" (UID: \"b490df30-7bfc-4c25-a9f5-768733b3b6e6\") " pod="openshift-network-operator/iptables-alerter-rb97g" Nov 01 06:09:51.447822 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447570 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-sys-fs\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:51.447822 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447601 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:51.447822 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447644 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr4ql\" (UniqueName: \"kubernetes.io/projected/93f3367a-3295-4f1c-931a-ba9c643d3b76-kube-api-access-xr4ql\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.447822 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447672 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57g2w\" (UniqueName: \"kubernetes.io/projected/ff209546-604b-462a-8f67-dee8fc8698d3-kube-api-access-57g2w\") pod \"service-ca-5fffc97bdf-ssrz5\" (UID: \"ff209546-604b-462a-8f67-dee8fc8698d3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ssrz5" Nov 01 06:09:51.447822 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447753 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b623310-9483-47a2-a5de-f19b87819eda-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:51.447822 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447799 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-run-ovn-kubernetes\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.448180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447859 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b490df30-7bfc-4c25-a9f5-768733b3b6e6-iptables-alerter-script\") pod \"iptables-alerter-rb97g\" (UID: \"b490df30-7bfc-4c25-a9f5-768733b3b6e6\") " pod="openshift-network-operator/iptables-alerter-rb97g" Nov 01 06:09:51.448180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447898 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-sysctl-conf\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.448180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447919 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ff209546-604b-462a-8f67-dee8fc8698d3-signing-key\") pod \"service-ca-5fffc97bdf-ssrz5\" (UID: \"ff209546-604b-462a-8f67-dee8fc8698d3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ssrz5" Nov 01 06:09:51.448180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.447956 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b623310-9483-47a2-a5de-f19b87819eda-system-cni-dir\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:51.448180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448018 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-daemon-config\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.448180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448066 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-run-netns\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.448180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448081 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-sys\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.448180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448096 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-lib-modules\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.448180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448114 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d48e417b-1b11-4ee3-8b45-0a71f68459bb-cni-binary-copy\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.448180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448128 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-run-netns\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.448180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448142 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-slash\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.448180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448179 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-etc-selinux\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:51.448564 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448223 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wqjh\" (UniqueName: \"kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh\") pod \"network-check-target-vvpd2\" (UID: \"6228c388-659f-4ca7-8d51-4fa5658424e2\") " pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:51.448564 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448272 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-etc-kubernetes\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.448564 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448303 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-cni-bin\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.448564 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448331 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.448564 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448361 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:51.448564 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448443 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/199a6bc2-3668-415c-8cc4-813f96eccfa4-agent-certs\") pod \"konnectivity-agent-h8j7v\" (UID: \"199a6bc2-3668-415c-8cc4-813f96eccfa4\") " pod="kube-system/konnectivity-agent-h8j7v" Nov 01 06:09:51.448564 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448468 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b856c58-9b9e-47b3-947a-ed56ae892e9b-host\") pod \"node-ca-vth7q\" (UID: \"7b856c58-9b9e-47b3-947a-ed56ae892e9b\") " pod="openshift-image-registry/node-ca-vth7q" Nov 01 06:09:51.448564 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448490 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-run\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.448564 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448514 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-var-lib-kubelet\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.448564 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448546 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-var-lib-cni-multus\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.448912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448577 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tjfc\" (UniqueName: \"kubernetes.io/projected/d48e417b-1b11-4ee3-8b45-0a71f68459bb-kube-api-access-4tjfc\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.448912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448605 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-hostroot\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.448912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448697 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-run-multus-certs\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.448912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448715 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs\") pod \"network-metrics-daemon-b98x2\" (UID: \"8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8\") " pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:51.448912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448730 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-conf-dir\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.448912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448755 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-crio-socket\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:51.448912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448787 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-systemd-units\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.448912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448812 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/24ddad96-0a15-4c10-b302-416833e87746-ovnkube-script-lib\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.448912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448846 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b0855c2-93a3-44a5-9311-63d891705e0e-kubelet-config\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:51.448912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448890 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mqsk\" (UniqueName: \"kubernetes.io/projected/568704c3-e334-44e1-8373-12a873c305b5-kube-api-access-6mqsk\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:51.448912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448909 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-socket-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:51.449242 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448966 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-data-volume\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:51.449242 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.448983 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-etc-openvswitch\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.449242 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.449018 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/24ddad96-0a15-4c10-b302-416833e87746-ovn-node-metrics-cert\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.449242 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.449033 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-data-volume\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:51.476022 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.475989 4385 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 06:09:51.476122 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:51.476050 4385 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:51.476934 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.476918 4385 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 06:09:51.476970 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:51.476959 4385 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-68.us-west-1.compute.internal" Nov 01 06:09:51.491386 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491348 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-78q7p\"" Nov 01 06:09:51.491386 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491371 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wss87\"" Nov 01 06:09:51.491556 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491410 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hnk2x\"" Nov 01 06:09:51.491556 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491443 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 06:09:51.491556 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491461 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 06:09:51.491556 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491469 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 06:09:51.491556 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491478 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vsdrw\"" Nov 01 06:09:51.491556 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491485 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 06:09:51.491556 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491489 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 06:09:51.491916 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491595 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 06:09:51.491916 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491626 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 06:09:51.491916 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491638 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 06:09:51.491916 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491664 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 06:09:51.491916 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491652 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 06:09:51.491916 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491672 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sdp5j\"" Nov 01 06:09:51.491916 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491683 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 06:09:51.491916 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491689 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 06:09:51.491916 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491798 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 06:09:51.491916 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491818 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 06:09:51.491916 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491881 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 06:09:51.491916 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491902 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 06:09:51.492254 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491984 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 06:09:51.492254 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.491995 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 06:09:51.492254 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.492220 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vwtj8\"" Nov 01 06:09:51.492327 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.492275 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 06:09:51.492327 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.492324 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 06:09:51.492417 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.492406 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 06:09:51.492448 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.492417 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 06:09:51.492448 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.492442 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 06:09:51.492502 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.492495 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 06:09:51.492530 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.492496 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 06:09:51.492560 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.492543 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rnlbn\"" Nov 01 06:09:51.492588 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.492582 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 06:09:51.492648 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.492618 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bnjvc\"" Nov 01 06:09:51.496989 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.496966 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/24ddad96-0a15-4c10-b302-416833e87746-env-overrides\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.497096 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.497015 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/24ddad96-0a15-4c10-b302-416833e87746-ovnkube-config\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.497096 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.497017 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b856c58-9b9e-47b3-947a-ed56ae892e9b-serviceca\") pod \"node-ca-vth7q\" (UID: \"7b856c58-9b9e-47b3-947a-ed56ae892e9b\") " pod="openshift-image-registry/node-ca-vth7q" Nov 01 06:09:51.497096 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.497038 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b623310-9483-47a2-a5de-f19b87819eda-cni-binary-copy\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:51.498215 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.498191 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b623310-9483-47a2-a5de-f19b87819eda-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:51.498319 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.498221 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b623310-9483-47a2-a5de-f19b87819eda-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:51.498319 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.498237 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/199a6bc2-3668-415c-8cc4-813f96eccfa4-konnectivity-ca\") pod \"konnectivity-agent-h8j7v\" (UID: \"199a6bc2-3668-415c-8cc4-813f96eccfa4\") " pod="kube-system/konnectivity-agent-h8j7v" Nov 01 06:09:51.498319 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.498237 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b490df30-7bfc-4c25-a9f5-768733b3b6e6-iptables-alerter-script\") pod \"iptables-alerter-rb97g\" (UID: \"b490df30-7bfc-4c25-a9f5-768733b3b6e6\") " pod="openshift-network-operator/iptables-alerter-rb97g" Nov 01 06:09:51.498319 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.498276 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-daemon-config\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.498447 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.498327 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d48e417b-1b11-4ee3-8b45-0a71f68459bb-cni-binary-copy\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.499410 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.499397 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/24ddad96-0a15-4c10-b302-416833e87746-ovnkube-script-lib\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.499560 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.499541 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/199a6bc2-3668-415c-8cc4-813f96eccfa4-agent-certs\") pod \"konnectivity-agent-h8j7v\" (UID: \"199a6bc2-3668-415c-8cc4-813f96eccfa4\") " pod="kube-system/konnectivity-agent-h8j7v" Nov 01 06:09:51.499635 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.499621 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/24ddad96-0a15-4c10-b302-416833e87746-ovn-node-metrics-cert\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.518962 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.518925 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 01 06:09:51.518962 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.518959 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 06:09:51.519407 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.519393 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8r6tp\"" Nov 01 06:09:51.519439 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.519393 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 06:09:51.525811 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.525794 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 06:09:51.525811 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.525802 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 06:09:51.525964 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.525818 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 01 06:09:51.525964 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.525827 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 06:09:51.525964 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.525851 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 01 06:09:51.525964 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.525893 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 06:09:51.526112 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.526097 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ff209546-604b-462a-8f67-dee8fc8698d3-signing-cabundle\") pod \"service-ca-5fffc97bdf-ssrz5\" (UID: \"ff209546-604b-462a-8f67-dee8fc8698d3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ssrz5" Nov 01 06:09:51.526824 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.526810 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6m98z\"" Nov 01 06:09:51.528019 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.528002 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:51.528185 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.528170 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ff209546-604b-462a-8f67-dee8fc8698d3-signing-key\") pod \"service-ca-5fffc97bdf-ssrz5\" (UID: \"ff209546-604b-462a-8f67-dee8fc8698d3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ssrz5" Nov 01 06:09:51.536371 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.536353 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 06:09:51.538441 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.538424 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 06:09:51.538441 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.538433 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8pggw\"" Nov 01 06:09:51.538566 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.538461 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jtkl2\"" Nov 01 06:09:51.538566 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.538462 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-h2lht\"" Nov 01 06:09:51.539718 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.539701 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 06:09:51.539782 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.539729 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 06:09:51.539782 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.539741 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 06:09:51.540331 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.540317 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 01 06:09:51.540496 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.540484 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jgqp9\"" Nov 01 06:09:51.540539 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.540527 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 06:09:51.540966 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.540949 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 06:09:51.540966 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.540960 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rlspf\"" Nov 01 06:09:51.541102 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.541063 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4w6wr\"" Nov 01 06:09:51.541696 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.541663 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhqdn\" (UniqueName: \"kubernetes.io/projected/3b623310-9483-47a2-a5de-f19b87819eda-kube-api-access-rhqdn\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:51.541696 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.541673 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5pnm\" (UniqueName: \"kubernetes.io/projected/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-kube-api-access-n5pnm\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:51.541899 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.541664 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb7kp\" (UniqueName: \"kubernetes.io/projected/b490df30-7bfc-4c25-a9f5-768733b3b6e6-kube-api-access-rb7kp\") pod \"iptables-alerter-rb97g\" (UID: \"b490df30-7bfc-4c25-a9f5-768733b3b6e6\") " pod="openshift-network-operator/iptables-alerter-rb97g" Nov 01 06:09:51.541899 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.541755 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 06:09:51.541899 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.541786 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z2gw\" (UniqueName: \"kubernetes.io/projected/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-kube-api-access-4z2gw\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:51.541899 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.541754 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr4ql\" (UniqueName: \"kubernetes.io/projected/93f3367a-3295-4f1c-931a-ba9c643d3b76-kube-api-access-xr4ql\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.542926 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.542201 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd7r9\" (UniqueName: \"kubernetes.io/projected/939039b6-e071-4a27-b24e-20bb2669dc15-kube-api-access-vd7r9\") pod \"node-resolver-ks27g\" (UID: \"939039b6-e071-4a27-b24e-20bb2669dc15\") " pod="openshift-dns/node-resolver-ks27g" Nov 01 06:09:51.542926 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.542581 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt9qk\" (UniqueName: \"kubernetes.io/projected/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-kube-api-access-qt9qk\") pod \"network-metrics-daemon-b98x2\" (UID: \"8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8\") " pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:51.542926 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.542692 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njlqm\" (UniqueName: \"kubernetes.io/projected/7b856c58-9b9e-47b3-947a-ed56ae892e9b-kube-api-access-njlqm\") pod \"node-ca-vth7q\" (UID: \"7b856c58-9b9e-47b3-947a-ed56ae892e9b\") " pod="openshift-image-registry/node-ca-vth7q" Nov 01 06:09:51.542926 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.542866 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tjfc\" (UniqueName: \"kubernetes.io/projected/d48e417b-1b11-4ee3-8b45-0a71f68459bb-kube-api-access-4tjfc\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.542926 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.542897 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vcgd\" (UniqueName: \"kubernetes.io/projected/24ddad96-0a15-4c10-b302-416833e87746-kube-api-access-4vcgd\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.543198 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.541948 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wqjh\" (UniqueName: \"kubernetes.io/projected/6228c388-659f-4ca7-8d51-4fa5658424e2-kube-api-access-6wqjh\") pod \"network-check-target-vvpd2\" (UID: \"6228c388-659f-4ca7-8d51-4fa5658424e2\") " pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:51.543198 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.542964 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mqsk\" (UniqueName: \"kubernetes.io/projected/568704c3-e334-44e1-8373-12a873c305b5-kube-api-access-6mqsk\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:51.543198 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.543101 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xd5q\" (UniqueName: \"kubernetes.io/projected/45331df5-857e-4663-9131-6575bcd3095a-kube-api-access-5xd5q\") pod \"ingress-canary-2vfjz\" (UID: \"45331df5-857e-4663-9131-6575bcd3095a\") " pod="openshift-ingress-canary/ingress-canary-2vfjz" Nov 01 06:09:51.545456 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.545434 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57g2w\" (UniqueName: \"kubernetes.io/projected/ff209546-604b-462a-8f67-dee8fc8698d3-kube-api-access-57g2w\") pod \"service-ca-5fffc97bdf-ssrz5\" (UID: \"ff209546-604b-462a-8f67-dee8fc8698d3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ssrz5" Nov 01 06:09:51.547003 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.546974 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/568704c3-e334-44e1-8373-12a873c305b5-config-volume\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:51.547600 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.547584 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45331df5-857e-4663-9131-6575bcd3095a-cert\") pod \"ingress-canary-2vfjz\" (UID: \"45331df5-857e-4663-9131-6575bcd3095a\") " pod="openshift-ingress-canary/ingress-canary-2vfjz" Nov 01 06:09:51.548743 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.548728 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b0855c2-93a3-44a5-9311-63d891705e0e-original-pull-secret\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:51.549867 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.549826 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/939039b6-e071-4a27-b24e-20bb2669dc15-hosts-file\") pod \"node-resolver-ks27g\" (UID: \"939039b6-e071-4a27-b24e-20bb2669dc15\") " pod="openshift-dns/node-resolver-ks27g" Nov 01 06:09:51.549939 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.549879 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-cnibin\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.549939 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.549903 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-cni-netd\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.549939 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.549926 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-system-cni-dir\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.550079 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.549948 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-log-socket\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.550079 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.549957 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-cnibin\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.550079 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.549973 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b0855c2-93a3-44a5-9311-63d891705e0e-dbus\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:51.550079 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.549947 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/939039b6-e071-4a27-b24e-20bb2669dc15-hosts-file\") pod \"node-resolver-ks27g\" (UID: \"939039b6-e071-4a27-b24e-20bb2669dc15\") " pod="openshift-dns/node-resolver-ks27g" Nov 01 06:09:51.550079 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.549980 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-cni-netd\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.550079 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550006 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-var-lib-openvswitch\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.550079 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550015 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-log-socket\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.550079 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550026 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-run-openvswitch\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.550079 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550046 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b0855c2-93a3-44a5-9311-63d891705e0e-dbus\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:51.550079 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550052 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-run-openvswitch\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.550079 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550058 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b490df30-7bfc-4c25-a9f5-768733b3b6e6-host-slash\") pod \"iptables-alerter-rb97g\" (UID: \"b490df30-7bfc-4c25-a9f5-768733b3b6e6\") " pod="openshift-network-operator/iptables-alerter-rb97g" Nov 01 06:09:51.550079 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550057 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-system-cni-dir\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.550079 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550088 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b490df30-7bfc-4c25-a9f5-768733b3b6e6-host-slash\") pod \"iptables-alerter-rb97g\" (UID: \"b490df30-7bfc-4c25-a9f5-768733b3b6e6\") " pod="openshift-network-operator/iptables-alerter-rb97g" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550112 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-var-lib-openvswitch\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550134 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-sys-fs\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550158 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550175 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-run-ovn-kubernetes\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550189 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-sysctl-conf\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550209 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b623310-9483-47a2-a5de-f19b87819eda-system-cni-dir\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550233 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-run-netns\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550242 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550256 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-sys\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550256 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-run-ovn-kubernetes\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550277 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-lib-modules\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550277 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-sys-fs\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550293 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-run-netns\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550307 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-slash\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550315 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-sys\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550323 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-etc-selinux\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550350 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b623310-9483-47a2-a5de-f19b87819eda-system-cni-dir\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550348 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-etc-kubernetes\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550363 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-slash\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550373 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-sysctl-conf\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550394 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-lib-modules\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550398 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-run-netns\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550404 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-etc-kubernetes\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550419 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-cni-bin\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550438 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-run-netns\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550444 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-cni-bin\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550450 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-etc-selinux\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550454 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550477 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b856c58-9b9e-47b3-947a-ed56ae892e9b-host\") pod \"node-ca-vth7q\" (UID: \"7b856c58-9b9e-47b3-947a-ed56ae892e9b\") " pod="openshift-image-registry/node-ca-vth7q" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550491 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-run\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550505 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-var-lib-kubelet\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550546 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550553 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-run\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550559 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-var-lib-kubelet\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.550563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550583 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b856c58-9b9e-47b3-947a-ed56ae892e9b-host\") pod \"node-ca-vth7q\" (UID: \"7b856c58-9b9e-47b3-947a-ed56ae892e9b\") " pod="openshift-image-registry/node-ca-vth7q" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550584 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-var-lib-cni-multus\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550608 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-hostroot\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550618 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-var-lib-cni-multus\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550625 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-run-multus-certs\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550646 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs\") pod \"network-metrics-daemon-b98x2\" (UID: \"8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8\") " pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550656 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-hostroot\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550669 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-conf-dir\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550692 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-crio-socket\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550699 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-conf-dir\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550669 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-run-multus-certs\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550716 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-systemd-units\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550737 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b0855c2-93a3-44a5-9311-63d891705e0e-kubelet-config\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550752 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-socket-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550766 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-etc-openvswitch\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550766 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-crio-socket\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550766 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-systemd-units\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550785 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-registration-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550811 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b0855c2-93a3-44a5-9311-63d891705e0e-kubelet-config\") pod \"global-pull-secret-syncer-txvqp\" (UID: \"5b0855c2-93a3-44a5-9311-63d891705e0e\") " pod="kube-system/global-pull-secret-syncer-txvqp" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550815 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-device-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550827 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-socket-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550824 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-etc-openvswitch\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550868 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-modprobe-d\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550877 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-device-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550884 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-var-lib-kubelet\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550901 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b623310-9483-47a2-a5de-f19b87819eda-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550924 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-socket-dir-parent\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550899 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e17cd4c-13b9-4dbf-8798-5fc68eaa1271-registration-dir\") pod \"aws-ebs-csi-driver-node-6gmfb\" (UID: \"1e17cd4c-13b9-4dbf-8798-5fc68eaa1271\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6gmfb" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550944 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550936 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-var-lib-kubelet\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550963 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-modprobe-d\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550971 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-run-systemd\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550994 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-run-systemd\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550996 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-socket-dir-parent\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550996 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-run-ovn\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551027 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-run-ovn\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551010 4385 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.550969 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b623310-9483-47a2-a5de-f19b87819eda-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551037 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-sysctl-d\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551073 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-cni-dir\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:51.551053 4385 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551120 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-run-k8s-cni-cncf-io\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551129 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-sysctl-d\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551092 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-run-k8s-cni-cncf-io\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:51.551162 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls podName:36effec9-edbb-40a4-8fe6-bee6b5bde3f1 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:52.051146656 +0000 UTC m=+1.814946508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b9rlh" (UID: "36effec9-edbb-40a4-8fe6-bee6b5bde3f1") : secret "insights-runtime-extractor-tls" not found Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551177 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-multus-cni-dir\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551182 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-var-lib-cni-bin\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551208 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d48e417b-1b11-4ee3-8b45-0a71f68459bb-host-var-lib-cni-bin\") pod \"multus-sxts6\" (UID: \"d48e417b-1b11-4ee3-8b45-0a71f68459bb\") " pod="openshift-multus/multus-sxts6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551224 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-kubelet\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551252 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-kubernetes\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551276 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-systemd\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551276 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-host-kubelet\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551307 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-host\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551322 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-kubernetes\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551331 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b623310-9483-47a2-a5de-f19b87819eda-cnibin\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551340 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-systemd\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551356 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-node-log\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551373 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-host\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551380 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b623310-9483-47a2-a5de-f19b87819eda-cnibin\") pod \"multus-additional-cni-plugins-fvnw6\" (UID: \"3b623310-9483-47a2-a5de-f19b87819eda\") " pod="openshift-multus/multus-additional-cni-plugins-fvnw6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551387 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551410 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-sysconfig\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551418 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/24ddad96-0a15-4c10-b302-416833e87746-node-log\") pod \"ovnkube-node-wj2f6\" (UID: \"24ddad96-0a15-4c10-b302-416833e87746\") " pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:51.551459 4385 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:51.551513 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls podName:568704c3-e334-44e1-8373-12a873c305b5 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:52.051497717 +0000 UTC m=+1.815297575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls") pod "dns-default-62hgp" (UID: "568704c3-e334-44e1-8373-12a873c305b5") : secret "dns-default-metrics-tls" not found Nov 01 06:09:51.551827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.551461 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/93f3367a-3295-4f1c-931a-ba9c643d3b76-etc-sysconfig\") pod \"tuned-258pf\" (UID: \"93f3367a-3295-4f1c-931a-ba9c643d3b76\") " pod="openshift-cluster-node-tuning-operator/tuned-258pf" Nov 01 06:09:51.554164 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.553207 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8-metrics-certs\") pod \"network-metrics-daemon-b98x2\" (UID: \"8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8\") " pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:51.656109 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.656013 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:51.741376 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.741346 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b98x2" Nov 01 06:09:51.802472 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.802440 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:51.952621 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:51.952541 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:52.054975 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:52.054936 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:52.055171 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:52.054990 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:52.055171 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:52.055087 4385 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 06:09:52.055171 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:52.055141 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls podName:36effec9-edbb-40a4-8fe6-bee6b5bde3f1 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:53.055126274 +0000 UTC m=+2.818926109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b9rlh" (UID: "36effec9-edbb-40a4-8fe6-bee6b5bde3f1") : secret "insights-runtime-extractor-tls" not found Nov 01 06:09:52.055171 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:52.055087 4385 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 06:09:52.055373 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:52.055218 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls podName:568704c3-e334-44e1-8373-12a873c305b5 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:53.055207746 +0000 UTC m=+2.819007598 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls") pod "dns-default-62hgp" (UID: "568704c3-e334-44e1-8373-12a873c305b5") : secret "dns-default-metrics-tls" not found Nov 01 06:09:52.093815 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:52.093782 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:52.139658 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:52.139624 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b98x2"] Nov 01 06:09:52.143740 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:52.143687 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8298b201_32cd_46f0_9dbf_fe0fe1c2b0c8.slice/crio-ed419911fe59c2f245fb25f721e45f12af0bd60725d5b2b9f483bfb239fc7d53 WatchSource:0}: Error finding container ed419911fe59c2f245fb25f721e45f12af0bd60725d5b2b9f483bfb239fc7d53: Status 404 returned error can't find the container with id ed419911fe59c2f245fb25f721e45f12af0bd60725d5b2b9f483bfb239fc7d53 Nov 01 06:09:52.145906 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:52.145888 4385 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:09:52.304677 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:52.304643 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:52.419815 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:52.419775 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b98x2" event={"ID":"8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8","Type":"ContainerStarted","Data":"ed419911fe59c2f245fb25f721e45f12af0bd60725d5b2b9f483bfb239fc7d53"} Nov 01 06:09:53.064224 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:53.064182 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:53.064386 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:53.064237 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:53.064386 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:53.064375 4385 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 06:09:53.064478 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:53.064382 4385 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 06:09:53.064478 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:53.064451 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls podName:568704c3-e334-44e1-8373-12a873c305b5 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:55.064435288 +0000 UTC m=+4.828235125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls") pod "dns-default-62hgp" (UID: "568704c3-e334-44e1-8373-12a873c305b5") : secret "dns-default-metrics-tls" not found Nov 01 06:09:53.064478 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:53.064465 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls podName:36effec9-edbb-40a4-8fe6-bee6b5bde3f1 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:55.064459355 +0000 UTC m=+4.828259191 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b9rlh" (UID: "36effec9-edbb-40a4-8fe6-bee6b5bde3f1") : secret "insights-runtime-extractor-tls" not found Nov 01 06:09:53.284541 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:53.284505 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:53.497027 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:53.496988 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:53.640143 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:53.640108 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:53.788685 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:53.788571 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:53.924208 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:53.924182 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:53.938210 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:53.938176 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:54.106481 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:54.106448 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:54.305549 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:54.305464 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:54.427285 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:54.427245 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b98x2" event={"ID":"8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8","Type":"ContainerStarted","Data":"22af894bad976883a0310f8690b7bc26399138987c29a7dce5494a1d16ce4701"} Nov 01 06:09:54.427450 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:54.427308 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b98x2" event={"ID":"8298b201-32cd-46f0-9dbf-fe0fe1c2b0c8","Type":"ContainerStarted","Data":"288d154312f835fcf2d9d5a31fd42f358a56807857b472a68ab990e2d1853593"} Nov 01 06:09:54.639497 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:54.639460 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:55.078672 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:55.078633 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:55.078864 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:55.078678 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:55.078864 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:55.078784 4385 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 06:09:55.078864 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:55.078786 4385 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 06:09:55.078864 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:55.078854 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls podName:568704c3-e334-44e1-8373-12a873c305b5 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:59.078825365 +0000 UTC m=+8.842625200 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls") pod "dns-default-62hgp" (UID: "568704c3-e334-44e1-8373-12a873c305b5") : secret "dns-default-metrics-tls" not found Nov 01 06:09:55.079076 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:09:55.078870 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls podName:36effec9-edbb-40a4-8fe6-bee6b5bde3f1 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:59.078864279 +0000 UTC m=+8.842664113 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b9rlh" (UID: "36effec9-edbb-40a4-8fe6-bee6b5bde3f1") : secret "insights-runtime-extractor-tls" not found Nov 01 06:09:55.290691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:55.290657 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:09:55.501369 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:55.501274 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:55.580084 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:55.580057 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:09:55.580269 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:55.580256 4385 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 06:09:55.695431 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:55.695399 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:55.901717 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:55.901675 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:56.078126 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:56.078092 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:56.216876 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:56.216761 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:56.359932 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:56.359901 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:57.502375 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:57.502338 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:57.696783 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:57.696742 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:57.837534 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:57.837485 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:57.989710 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:57.989678 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:58.133333 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:58.133238 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:58.302240 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:58.302208 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:58.705897 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:58.705860 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:59.106082 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:59.106047 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:59.106250 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:59.106090 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:59.108536 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:59.108510 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36effec9-edbb-40a4-8fe6-bee6b5bde3f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9rlh\" (UID: \"36effec9-edbb-40a4-8fe6-bee6b5bde3f1\") " pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:59.108635 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:59.108539 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/568704c3-e334-44e1-8373-12a873c305b5-metrics-tls\") pod \"dns-default-62hgp\" (UID: \"568704c3-e334-44e1-8373-12a873c305b5\") " pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:59.243684 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:59.243639 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-62hgp" Nov 01 06:09:59.243684 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:59.243667 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b9rlh" Nov 01 06:09:59.714171 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:59.714140 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b9rlh"] Nov 01 06:09:59.764002 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:09:59.763974 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-62hgp"] Nov 01 06:09:59.766611 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:59.766576 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36effec9_edbb_40a4_8fe6_bee6b5bde3f1.slice/crio-6a2e264431606812210cfa04e9635650493df577ed20029482d3a06afa074b91 WatchSource:0}: Error finding container 6a2e264431606812210cfa04e9635650493df577ed20029482d3a06afa074b91: Status 404 returned error can't find the container with id 6a2e264431606812210cfa04e9635650493df577ed20029482d3a06afa074b91 Nov 01 06:09:59.767254 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:09:59.767220 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568704c3_e334_44e1_8373_12a873c305b5.slice/crio-455070a9cee275f7a9c54c29e78722f30c544c19fcd2c706968f57c0c0ed8e7c WatchSource:0}: Error finding container 455070a9cee275f7a9c54c29e78722f30c544c19fcd2c706968f57c0c0ed8e7c: Status 404 returned error can't find the container with id 455070a9cee275f7a9c54c29e78722f30c544c19fcd2c706968f57c0c0ed8e7c Nov 01 06:10:00.143881 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.143825 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wj2f6" Nov 01 06:10:00.314732 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.314697 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6qjdb"] Nov 01 06:10:00.316416 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.316401 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6qjdb" Nov 01 06:10:00.377076 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.377045 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 01 06:10:00.377076 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.377051 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-4p2ck\"" Nov 01 06:10:00.404857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.404745 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54c856867d-jswct"] Nov 01 06:10:00.406755 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.406730 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.437622 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.437514 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/703b9839-6695-4c5a-a243-77a2f350da29-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-6qjdb\" (UID: \"703b9839-6695-4c5a-a243-77a2f350da29\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6qjdb" Nov 01 06:10:00.439914 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.439885 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:00.453880 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.451676 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9rlh" event={"ID":"36effec9-edbb-40a4-8fe6-bee6b5bde3f1","Type":"ContainerStarted","Data":"5bfc003f594578810364fa9fe98e917b568195a45c7e3f395b16d6657874ca02"} Nov 01 06:10:00.453880 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.451780 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9rlh" event={"ID":"36effec9-edbb-40a4-8fe6-bee6b5bde3f1","Type":"ContainerStarted","Data":"6a2e264431606812210cfa04e9635650493df577ed20029482d3a06afa074b91"} Nov 01 06:10:00.453880 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.452971 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-62hgp" event={"ID":"568704c3-e334-44e1-8373-12a873c305b5","Type":"ContainerStarted","Data":"455070a9cee275f7a9c54c29e78722f30c544c19fcd2c706968f57c0c0ed8e7c"} Nov 01 06:10:00.463520 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.463488 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 01 06:10:00.476750 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.476716 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kqrlq\"" Nov 01 06:10:00.476750 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.476727 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 01 06:10:00.476985 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.476809 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 01 06:10:00.476985 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.476812 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 01 06:10:00.479293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.479266 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54c856867d-jswct"] Nov 01 06:10:00.538620 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.538573 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a8ea1cd-0445-4005-b339-07a51ac7238a-registry-certificates\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.538819 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.538632 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a8ea1cd-0445-4005-b339-07a51ac7238a-ca-trust-extracted\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.538819 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.538674 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/703b9839-6695-4c5a-a243-77a2f350da29-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-6qjdb\" (UID: \"703b9839-6695-4c5a-a243-77a2f350da29\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6qjdb" Nov 01 06:10:00.538819 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.538753 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a8ea1cd-0445-4005-b339-07a51ac7238a-registry-tls\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.538819 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.538781 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a8ea1cd-0445-4005-b339-07a51ac7238a-bound-sa-token\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.539021 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.538914 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jtp4\" (UniqueName: \"kubernetes.io/projected/3a8ea1cd-0445-4005-b339-07a51ac7238a-kube-api-access-5jtp4\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.539021 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.538975 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a8ea1cd-0445-4005-b339-07a51ac7238a-image-registry-private-configuration\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.539021 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.539007 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a8ea1cd-0445-4005-b339-07a51ac7238a-trusted-ca\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.539124 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.539030 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a8ea1cd-0445-4005-b339-07a51ac7238a-installation-pull-secrets\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.539767 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.539738 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6qjdb"] Nov 01 06:10:00.541909 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.541885 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/703b9839-6695-4c5a-a243-77a2f350da29-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-6qjdb\" (UID: \"703b9839-6695-4c5a-a243-77a2f350da29\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6qjdb" Nov 01 06:10:00.543021 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.542994 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h8j7v" Nov 01 06:10:00.625441 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.625403 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:00.626149 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.626132 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6qjdb" Nov 01 06:10:00.639801 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.639769 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a8ea1cd-0445-4005-b339-07a51ac7238a-image-registry-private-configuration\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.639953 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.639813 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a8ea1cd-0445-4005-b339-07a51ac7238a-trusted-ca\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.639953 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.639865 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a8ea1cd-0445-4005-b339-07a51ac7238a-installation-pull-secrets\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.639953 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.639893 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a8ea1cd-0445-4005-b339-07a51ac7238a-registry-certificates\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.639953 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.639929 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a8ea1cd-0445-4005-b339-07a51ac7238a-ca-trust-extracted\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.640135 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.640098 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a8ea1cd-0445-4005-b339-07a51ac7238a-registry-tls\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.640195 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.640135 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a8ea1cd-0445-4005-b339-07a51ac7238a-bound-sa-token\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.640195 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.640178 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jtp4\" (UniqueName: \"kubernetes.io/projected/3a8ea1cd-0445-4005-b339-07a51ac7238a-kube-api-access-5jtp4\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.640828 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.640805 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a8ea1cd-0445-4005-b339-07a51ac7238a-ca-trust-extracted\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.641348 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.641318 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a8ea1cd-0445-4005-b339-07a51ac7238a-registry-certificates\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.641571 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.641548 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a8ea1cd-0445-4005-b339-07a51ac7238a-trusted-ca\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.642982 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.642932 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a8ea1cd-0445-4005-b339-07a51ac7238a-image-registry-private-configuration\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.643489 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.643464 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a8ea1cd-0445-4005-b339-07a51ac7238a-registry-tls\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.643684 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.643667 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a8ea1cd-0445-4005-b339-07a51ac7238a-installation-pull-secrets\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.697482 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.697454 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a8ea1cd-0445-4005-b339-07a51ac7238a-bound-sa-token\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.697682 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.697659 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jtp4\" (UniqueName: \"kubernetes.io/projected/3a8ea1cd-0445-4005-b339-07a51ac7238a-kube-api-access-5jtp4\") pod \"image-registry-54c856867d-jswct\" (UID: \"3a8ea1cd-0445-4005-b339-07a51ac7238a\") " pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.742106 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.742072 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:00.782379 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.782337 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:00.872622 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.872587 4385 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-68.us-west-1.compute.internal" event="NodeReady" Nov 01 06:10:00.960050 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.960003 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:00.992466 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:00.992430 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6qjdb"] Nov 01 06:10:00.997788 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:10:00.997750 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703b9839_6695_4c5a_a243_77a2f350da29.slice/crio-09496da6d8797e8808a0b4ba2f26b5965fd0945a9a3a7d1592f307d5b2b64f81 WatchSource:0}: Error finding container 09496da6d8797e8808a0b4ba2f26b5965fd0945a9a3a7d1592f307d5b2b64f81: Status 404 returned error can't find the container with id 09496da6d8797e8808a0b4ba2f26b5965fd0945a9a3a7d1592f307d5b2b64f81 Nov 01 06:10:01.114042 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.114003 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54c856867d-jswct"] Nov 01 06:10:01.122346 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:10:01.122303 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8ea1cd_0445_4005_b339_07a51ac7238a.slice/crio-fdb5b8a4cf77622b3f8500475aca2a4c3bb5a9abc6a35e12031f3c3abe4073ae WatchSource:0}: Error finding container fdb5b8a4cf77622b3f8500475aca2a4c3bb5a9abc6a35e12031f3c3abe4073ae: Status 404 returned error can't find the container with id fdb5b8a4cf77622b3f8500475aca2a4c3bb5a9abc6a35e12031f3c3abe4073ae Nov 01 06:10:01.122552 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.122521 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:01.269914 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.269879 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:01.425295 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.425259 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:01.432066 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.432036 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-lsfzp"] Nov 01 06:10:01.434026 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.434000 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-lsfzp" Nov 01 06:10:01.459436 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.459389 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9rlh" event={"ID":"36effec9-edbb-40a4-8fe6-bee6b5bde3f1","Type":"ContainerStarted","Data":"758a8891728638b2ed967f6e111e3624340c2c391a2510e72749e8dd61cef3cb"} Nov 01 06:10:01.461520 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.461471 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54c856867d-jswct" event={"ID":"3a8ea1cd-0445-4005-b339-07a51ac7238a","Type":"ContainerStarted","Data":"0e68a806bb14c86f4e50311c2bed4a28e8e93393ecea37947308950baa96c557"} Nov 01 06:10:01.461652 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.461531 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54c856867d-jswct" event={"ID":"3a8ea1cd-0445-4005-b339-07a51ac7238a","Type":"ContainerStarted","Data":"fdb5b8a4cf77622b3f8500475aca2a4c3bb5a9abc6a35e12031f3c3abe4073ae"} Nov 01 06:10:01.462175 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.462149 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:01.463369 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.463335 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6qjdb" event={"ID":"703b9839-6695-4c5a-a243-77a2f350da29","Type":"ContainerStarted","Data":"09496da6d8797e8808a0b4ba2f26b5965fd0945a9a3a7d1592f307d5b2b64f81"} Nov 01 06:10:01.486001 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.485966 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 06:10:01.486001 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.485980 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 06:10:01.499474 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.499442 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-lsfzp"] Nov 01 06:10:01.547804 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.547296 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twl6w\" (UniqueName: \"kubernetes.io/projected/1468808d-a757-4433-8557-2458e1a6ed03-kube-api-access-twl6w\") pod \"downloads-6dd789c8b7-lsfzp\" (UID: \"1468808d-a757-4433-8557-2458e1a6ed03\") " pod="openshift-console/downloads-6dd789c8b7-lsfzp" Nov 01 06:10:01.547804 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.547397 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1468808d-a757-4433-8557-2458e1a6ed03-tmp\") pod \"downloads-6dd789c8b7-lsfzp\" (UID: \"1468808d-a757-4433-8557-2458e1a6ed03\") " pod="openshift-console/downloads-6dd789c8b7-lsfzp" Nov 01 06:10:01.580609 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.580563 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:01.647935 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.647898 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twl6w\" (UniqueName: \"kubernetes.io/projected/1468808d-a757-4433-8557-2458e1a6ed03-kube-api-access-twl6w\") pod \"downloads-6dd789c8b7-lsfzp\" (UID: \"1468808d-a757-4433-8557-2458e1a6ed03\") " pod="openshift-console/downloads-6dd789c8b7-lsfzp" Nov 01 06:10:01.648128 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.647942 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1468808d-a757-4433-8557-2458e1a6ed03-tmp\") pod \"downloads-6dd789c8b7-lsfzp\" (UID: \"1468808d-a757-4433-8557-2458e1a6ed03\") " pod="openshift-console/downloads-6dd789c8b7-lsfzp" Nov 01 06:10:01.648413 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.648381 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1468808d-a757-4433-8557-2458e1a6ed03-tmp\") pod \"downloads-6dd789c8b7-lsfzp\" (UID: \"1468808d-a757-4433-8557-2458e1a6ed03\") " pod="openshift-console/downloads-6dd789c8b7-lsfzp" Nov 01 06:10:01.705157 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.705084 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twl6w\" (UniqueName: \"kubernetes.io/projected/1468808d-a757-4433-8557-2458e1a6ed03-kube-api-access-twl6w\") pod \"downloads-6dd789c8b7-lsfzp\" (UID: \"1468808d-a757-4433-8557-2458e1a6ed03\") " pod="openshift-console/downloads-6dd789c8b7-lsfzp" Nov 01 06:10:01.751867 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.746818 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-lsfzp" Nov 01 06:10:01.785355 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:01.781580 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:02.129656 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:02.129621 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-lsfzp"] Nov 01 06:10:02.133991 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:10:02.133956 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1468808d_a757_4433_8557_2458e1a6ed03.slice/crio-15e2c70de016c403688ae977e6db1ec158de7241cdbcf1cf8e621915cbde9c1b WatchSource:0}: Error finding container 15e2c70de016c403688ae977e6db1ec158de7241cdbcf1cf8e621915cbde9c1b: Status 404 returned error can't find the container with id 15e2c70de016c403688ae977e6db1ec158de7241cdbcf1cf8e621915cbde9c1b Nov 01 06:10:02.174428 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:02.174381 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:02.280738 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:02.280686 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-54c856867d-jswct" podStartSLOduration=2.280671446 podStartE2EDuration="2.280671446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:02.279718085 +0000 UTC m=+12.043517941" watchObservedRunningTime="2026-11-01 06:10:02.280671446 +0000 UTC m=+12.044471303" Nov 01 06:10:02.322181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:02.322140 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:02.468624 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:02.468495 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-lsfzp" event={"ID":"1468808d-a757-4433-8557-2458e1a6ed03","Type":"ContainerStarted","Data":"15e2c70de016c403688ae977e6db1ec158de7241cdbcf1cf8e621915cbde9c1b"} Nov 01 06:10:02.472213 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:02.472180 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-62hgp" event={"ID":"568704c3-e334-44e1-8373-12a873c305b5","Type":"ContainerStarted","Data":"fd6505951a1c5a32e483f3c4c006eb48831b46423e38f7f60421cd3839439d79"} Nov 01 06:10:02.472403 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:02.472231 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-62hgp" event={"ID":"568704c3-e334-44e1-8373-12a873c305b5","Type":"ContainerStarted","Data":"3c69df3813d86045c42ce7425e89075b890fda15c403b93d1f028a2f83561fb8"} Nov 01 06:10:02.472403 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:02.472269 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-62hgp" Nov 01 06:10:02.568026 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:02.567991 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:02.707467 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:02.707421 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:03.482659 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:03.482619 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6qjdb" event={"ID":"703b9839-6695-4c5a-a243-77a2f350da29","Type":"ContainerStarted","Data":"ba3460892fa4aff261ffba8bc6497974cae4d34908a982d33606efbca26c0133"} Nov 01 06:10:03.483249 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:03.483000 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6qjdb" Nov 01 06:10:03.485504 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:03.485458 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9rlh" event={"ID":"36effec9-edbb-40a4-8fe6-bee6b5bde3f1","Type":"ContainerStarted","Data":"638b46c7943c77c1b6660bf6aeacbd646859b12238b26f29cff0e0f839fd0556"} Nov 01 06:10:03.488396 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:03.488372 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6qjdb" Nov 01 06:10:03.575503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:03.575472 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:03.595727 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:03.595670 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6qjdb" podStartSLOduration=2.011891369 podStartE2EDuration="3.595655019s" totalImagesPullingTime="1.58376365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:03.594365831 +0000 UTC m=+13.358165689" watchObservedRunningTime="2026-11-01 06:10:03.595655019 +0000 UTC m=+13.359454877" Nov 01 06:10:03.734645 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:03.734570 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:03.916282 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:03.916244 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:04.065015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.064926 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:04.211345 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.211291 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:04.301124 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.301085 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-clhd7"] Nov 01 06:10:04.303089 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.303062 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-clhd7" Nov 01 06:10:04.355130 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.355080 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:04.355613 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.355573 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 06:10:04.355874 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.355853 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 06:10:04.355874 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.355868 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 06:10:04.356071 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.355868 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 06:10:04.356071 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.355906 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nx7b8\"" Nov 01 06:10:04.356071 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.355965 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 06:10:04.362781 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.362749 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-clhd7"] Nov 01 06:10:04.369103 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.369080 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23904dd3-3c84-4aba-aad2-6db71df3eda1-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-clhd7\" (UID: \"23904dd3-3c84-4aba-aad2-6db71df3eda1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-clhd7" Nov 01 06:10:04.369222 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.369113 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq7m6\" (UniqueName: \"kubernetes.io/projected/23904dd3-3c84-4aba-aad2-6db71df3eda1-kube-api-access-mq7m6\") pod \"prometheus-operator-9cd89b74c-clhd7\" (UID: \"23904dd3-3c84-4aba-aad2-6db71df3eda1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-clhd7" Nov 01 06:10:04.369222 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.369206 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/23904dd3-3c84-4aba-aad2-6db71df3eda1-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-clhd7\" (UID: \"23904dd3-3c84-4aba-aad2-6db71df3eda1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-clhd7" Nov 01 06:10:04.369341 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.369232 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23904dd3-3c84-4aba-aad2-6db71df3eda1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-clhd7\" (UID: \"23904dd3-3c84-4aba-aad2-6db71df3eda1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-clhd7" Nov 01 06:10:04.469562 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.469527 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq7m6\" (UniqueName: \"kubernetes.io/projected/23904dd3-3c84-4aba-aad2-6db71df3eda1-kube-api-access-mq7m6\") pod \"prometheus-operator-9cd89b74c-clhd7\" (UID: \"23904dd3-3c84-4aba-aad2-6db71df3eda1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-clhd7" Nov 01 06:10:04.469756 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.469592 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/23904dd3-3c84-4aba-aad2-6db71df3eda1-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-clhd7\" (UID: \"23904dd3-3c84-4aba-aad2-6db71df3eda1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-clhd7" Nov 01 06:10:04.469756 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.469625 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23904dd3-3c84-4aba-aad2-6db71df3eda1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-clhd7\" (UID: \"23904dd3-3c84-4aba-aad2-6db71df3eda1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-clhd7" Nov 01 06:10:04.469756 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.469668 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23904dd3-3c84-4aba-aad2-6db71df3eda1-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-clhd7\" (UID: \"23904dd3-3c84-4aba-aad2-6db71df3eda1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-clhd7" Nov 01 06:10:04.470663 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.470451 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23904dd3-3c84-4aba-aad2-6db71df3eda1-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-clhd7\" (UID: \"23904dd3-3c84-4aba-aad2-6db71df3eda1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-clhd7" Nov 01 06:10:04.472490 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.472462 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23904dd3-3c84-4aba-aad2-6db71df3eda1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-clhd7\" (UID: \"23904dd3-3c84-4aba-aad2-6db71df3eda1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-clhd7" Nov 01 06:10:04.472603 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.472552 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/23904dd3-3c84-4aba-aad2-6db71df3eda1-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-clhd7\" (UID: \"23904dd3-3c84-4aba-aad2-6db71df3eda1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-clhd7" Nov 01 06:10:04.526798 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.526771 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq7m6\" (UniqueName: \"kubernetes.io/projected/23904dd3-3c84-4aba-aad2-6db71df3eda1-kube-api-access-mq7m6\") pod \"prometheus-operator-9cd89b74c-clhd7\" (UID: \"23904dd3-3c84-4aba-aad2-6db71df3eda1\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-clhd7" Nov 01 06:10:04.615861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.615761 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-clhd7" Nov 01 06:10:04.992626 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:04.992536 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-clhd7"] Nov 01 06:10:05.029459 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:10:05.027213 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23904dd3_3c84_4aba_aad2_6db71df3eda1.slice/crio-2098fbcd9c14f259d345f96e59c06b7856b859c7da9e14b854b4b925763f357f WatchSource:0}: Error finding container 2098fbcd9c14f259d345f96e59c06b7856b859c7da9e14b854b4b925763f357f: Status 404 returned error can't find the container with id 2098fbcd9c14f259d345f96e59c06b7856b859c7da9e14b854b4b925763f357f Nov 01 06:10:05.175385 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:05.175346 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:05.466535 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:05.466491 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:05.494754 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:05.494718 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-clhd7" event={"ID":"23904dd3-3c84-4aba-aad2-6db71df3eda1","Type":"ContainerStarted","Data":"2098fbcd9c14f259d345f96e59c06b7856b859c7da9e14b854b4b925763f357f"} Nov 01 06:10:05.640389 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:05.640345 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:06.785227 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:10:06.785153 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:10:07.503814 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:07.503731 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-clhd7" event={"ID":"23904dd3-3c84-4aba-aad2-6db71df3eda1","Type":"ContainerStarted","Data":"327a52c5a3cbcea31b112e85fb019aaeb4ace20cf28354fb26b3e251b35572fd"} Nov 01 06:10:07.503814 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:07.503801 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-clhd7" event={"ID":"23904dd3-3c84-4aba-aad2-6db71df3eda1","Type":"ContainerStarted","Data":"78d9cd55c011bb5e19e25a4eaa80b7d3f047994dc78ada371e21fbcdf5a23a87"} Nov 01 06:10:07.616935 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:07.616879 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-clhd7" podStartSLOduration=1.538492877 podStartE2EDuration="3.616861488s" totalImagesPullingTime="2.078368611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:07.616269951 +0000 UTC m=+17.380069807" watchObservedRunningTime="2026-11-01 06:10:07.616861488 +0000 UTC m=+17.380661345" Nov 01 06:10:09.751460 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:09.751426 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:10.320807 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:10.320772 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:10.477476 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:10.477436 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:10.678782 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:10.678746 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:10.832060 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:10.832018 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:11.001235 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.001159 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tf6zg"] Nov 01 06:10:11.003778 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.003741 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.003935 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.003889 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:11.099205 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.099174 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 06:10:11.119848 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.119809 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/23c957b7-d233-4fbe-a14e-4098f4229462-node-exporter-accelerators-collector-config\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.120037 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.119883 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23c957b7-d233-4fbe-a14e-4098f4229462-sys\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.120037 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.119976 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxxnl\" (UniqueName: \"kubernetes.io/projected/23c957b7-d233-4fbe-a14e-4098f4229462-kube-api-access-lxxnl\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.120037 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.120031 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/23c957b7-d233-4fbe-a14e-4098f4229462-root\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.120349 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.120077 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/23c957b7-d233-4fbe-a14e-4098f4229462-node-exporter-tls\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.120349 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.120102 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/23c957b7-d233-4fbe-a14e-4098f4229462-node-exporter-textfile\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.120349 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.120131 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23c957b7-d233-4fbe-a14e-4098f4229462-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.120523 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.120393 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/23c957b7-d233-4fbe-a14e-4098f4229462-node-exporter-wtmp\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.120523 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.120417 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23c957b7-d233-4fbe-a14e-4098f4229462-metrics-client-ca\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.149140 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.149086 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 06:10:11.149140 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.149123 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qh54x\"" Nov 01 06:10:11.149140 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.149131 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 06:10:11.151787 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.151756 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:11.221755 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.221716 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23c957b7-d233-4fbe-a14e-4098f4229462-sys\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.221960 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.221765 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxxnl\" (UniqueName: \"kubernetes.io/projected/23c957b7-d233-4fbe-a14e-4098f4229462-kube-api-access-lxxnl\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.221960 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.221801 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/23c957b7-d233-4fbe-a14e-4098f4229462-root\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.221960 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.221850 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23c957b7-d233-4fbe-a14e-4098f4229462-sys\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.221960 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.221859 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/23c957b7-d233-4fbe-a14e-4098f4229462-node-exporter-tls\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.221960 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.221886 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/23c957b7-d233-4fbe-a14e-4098f4229462-node-exporter-textfile\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.221960 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.221909 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/23c957b7-d233-4fbe-a14e-4098f4229462-root\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.221960 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.221917 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23c957b7-d233-4fbe-a14e-4098f4229462-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.221960 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.221946 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/23c957b7-d233-4fbe-a14e-4098f4229462-node-exporter-wtmp\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.222244 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.221968 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23c957b7-d233-4fbe-a14e-4098f4229462-metrics-client-ca\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.222244 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.222019 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/23c957b7-d233-4fbe-a14e-4098f4229462-node-exporter-accelerators-collector-config\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.222325 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.222294 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/23c957b7-d233-4fbe-a14e-4098f4229462-node-exporter-textfile\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.222375 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.222316 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/23c957b7-d233-4fbe-a14e-4098f4229462-node-exporter-wtmp\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.222630 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.222607 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/23c957b7-d233-4fbe-a14e-4098f4229462-node-exporter-accelerators-collector-config\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.222761 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.222741 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23c957b7-d233-4fbe-a14e-4098f4229462-metrics-client-ca\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.225046 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.224717 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23c957b7-d233-4fbe-a14e-4098f4229462-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.225046 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.225001 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/23c957b7-d233-4fbe-a14e-4098f4229462-node-exporter-tls\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.279698 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.279618 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxxnl\" (UniqueName: \"kubernetes.io/projected/23c957b7-d233-4fbe-a14e-4098f4229462-kube-api-access-lxxnl\") pod \"node-exporter-tf6zg\" (UID: \"23c957b7-d233-4fbe-a14e-4098f4229462\") " pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.297092 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.297056 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:11.317992 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.317958 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tf6zg" Nov 01 06:10:11.329795 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:10:11.329761 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c957b7_d233_4fbe_a14e_4098f4229462.slice/crio-2bde14dea42a640a985596554d2df0ab5b5cac1a92864fd4563b24fe6a463023 WatchSource:0}: Error finding container 2bde14dea42a640a985596554d2df0ab5b5cac1a92864fd4563b24fe6a463023: Status 404 returned error can't find the container with id 2bde14dea42a640a985596554d2df0ab5b5cac1a92864fd4563b24fe6a463023 Nov 01 06:10:11.447191 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.447155 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:11.519761 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:11.519718 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tf6zg" event={"ID":"23c957b7-d233-4fbe-a14e-4098f4229462","Type":"ContainerStarted","Data":"2bde14dea42a640a985596554d2df0ab5b5cac1a92864fd4563b24fe6a463023"} Nov 01 06:10:12.323703 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:12.323666 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:12.653211 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:12.653174 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:12.992207 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:12.992176 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:13.144466 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.144426 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:13.322861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.322747 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:13.492746 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.492715 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-62hgp" Nov 01 06:10:13.531913 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.531879 4385 generic.go:356] "Generic (PLEG): container finished" podID="23c957b7-d233-4fbe-a14e-4098f4229462" containerID="e4f5e8863642fc8d2e659dbe16ea5a8aa17cd60843c37bbc8277974d12ab5a83" exitCode=0 Nov 01 06:10:13.532103 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.531933 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tf6zg" event={"ID":"23c957b7-d233-4fbe-a14e-4098f4229462","Type":"ContainerDied","Data":"e4f5e8863642fc8d2e659dbe16ea5a8aa17cd60843c37bbc8277974d12ab5a83"} Nov 01 06:10:13.596630 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.596599 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb"] Nov 01 06:10:13.598807 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.598783 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.648091 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.648066 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb"] Nov 01 06:10:13.653274 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.653230 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 01 06:10:13.656207 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.656041 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 01 06:10:13.656207 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.656094 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 01 06:10:13.656207 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.656138 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 01 06:10:13.656207 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.656148 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 01 06:10:13.656207 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.656175 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9jbpqt0t45s45\"" Nov 01 06:10:13.656207 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.656207 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-hkzfx\"" Nov 01 06:10:13.746121 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.746083 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.746320 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.746126 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.746320 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.746156 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.746320 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.746218 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-secret-grpc-tls\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.746463 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.746317 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b7wh\" (UniqueName: \"kubernetes.io/projected/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-kube-api-access-6b7wh\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.746463 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.746347 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-tmp\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.746463 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.746375 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-metrics-client-ca\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.746575 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.746479 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-secret-thanos-querier-tls\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.746575 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.746507 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.847259 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.847172 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-tmp\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.847259 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.847229 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-metrics-client-ca\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.847465 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.847285 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-secret-thanos-querier-tls\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.847465 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.847323 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.847465 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.847360 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.847465 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.847398 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.847465 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.847430 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.847465 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.847453 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-secret-grpc-tls\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.847668 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.847485 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b7wh\" (UniqueName: \"kubernetes.io/projected/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-kube-api-access-6b7wh\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.847668 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.847634 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-tmp\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.848067 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.848014 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-metrics-client-ca\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.850777 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.850751 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.850949 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.850881 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-secret-thanos-querier-tls\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.850949 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.850911 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.851050 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.850911 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.851965 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.851940 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.852401 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.852380 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-secret-grpc-tls\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.904108 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.904079 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b7wh\" (UniqueName: \"kubernetes.io/projected/3ddc5224-736b-4cc5-8ed1-1cd16f35fc61-kube-api-access-6b7wh\") pod \"thanos-querier-56b4f84ff8-qfxnb\" (UID: \"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61\") " pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:13.921702 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:13.921670 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:14.346948 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:14.346919 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb"] Nov 01 06:10:14.349433 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:10:14.349402 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ddc5224_736b_4cc5_8ed1_1cd16f35fc61.slice/crio-a57073945f9d32601569630c43fdc3e962907ea120f90b0472c3bf6b596cebe2 WatchSource:0}: Error finding container a57073945f9d32601569630c43fdc3e962907ea120f90b0472c3bf6b596cebe2: Status 404 returned error can't find the container with id a57073945f9d32601569630c43fdc3e962907ea120f90b0472c3bf6b596cebe2 Nov 01 06:10:14.538973 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:14.538920 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tf6zg" event={"ID":"23c957b7-d233-4fbe-a14e-4098f4229462","Type":"ContainerStarted","Data":"3482db983faf2ce03ba81c964d0837e7e046e7fe64578d85ac48a26f42ad7bab"} Nov 01 06:10:14.539484 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:14.539005 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tf6zg" event={"ID":"23c957b7-d233-4fbe-a14e-4098f4229462","Type":"ContainerStarted","Data":"093ac76a8b4f65584b6afc8fe9326b6e0b3acf9efca5e07a5a8ffc56d9ff57b0"} Nov 01 06:10:14.540558 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:14.540523 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" event={"ID":"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61","Type":"ContainerStarted","Data":"a57073945f9d32601569630c43fdc3e962907ea120f90b0472c3bf6b596cebe2"} Nov 01 06:10:14.653429 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:14.653333 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tf6zg" podStartSLOduration=3.094571229 podStartE2EDuration="4.653317912s" totalImagesPullingTime="1.470641335s" totalInitContainerRuntime="88.105348ms" isStatefulPod=true podCreationTimestamp="2026-11-01 06:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:14.652337754 +0000 UTC m=+24.416137611" watchObservedRunningTime="2026-11-01 06:10:14.653317912 +0000 UTC m=+24.417117815" Nov 01 06:10:15.071016 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:15.070933 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:15.218991 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:15.218959 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:15.368819 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:15.368782 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:15.510521 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:15.510463 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:15.655795 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:15.655697 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:16.103911 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:16.103805 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:16.280510 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:16.280470 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:16.418852 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:16.418762 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:16.571247 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:16.571164 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:16.730127 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:16.730039 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:16.878633 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:16.878592 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:17.554086 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:17.553992 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" event={"ID":"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61","Type":"ContainerStarted","Data":"0eefc41121c9ae546676981abac086ca61464e7c47ee73db5979b1ed088713df"} Nov 01 06:10:17.554247 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:17.554097 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" event={"ID":"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61","Type":"ContainerStarted","Data":"662c17f113290b377af47fa1783eeec22105b555d9d518dca8648b14056ebd44"} Nov 01 06:10:17.554247 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:17.554137 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" event={"ID":"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61","Type":"ContainerStarted","Data":"8b395e1aec3cf7a8dd1d48e88ef3b0a4aecc7886696257202e9e0ad0832e5e3c"} Nov 01 06:10:18.653598 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:18.653558 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:18.839495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:18.839456 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:19.461055 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:19.461023 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:19.565161 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:19.565075 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" event={"ID":"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61","Type":"ContainerStarted","Data":"d0ff0274094fee33023c58ba6e6f7897240a2286eef80b91556b761adce2f302"} Nov 01 06:10:19.565323 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:19.565209 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:19.565323 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:19.565257 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" event={"ID":"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61","Type":"ContainerStarted","Data":"c0e11cf3b862bce501119e8f927069e43b4cfd0bdb547d44119b995625cc652c"} Nov 01 06:10:19.565323 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:19.565293 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" event={"ID":"3ddc5224-736b-4cc5-8ed1-1cd16f35fc61","Type":"ContainerStarted","Data":"324270709c44ef727a66caf55263692dd0bf9a68256aecc54f880ae08c0340a8"} Nov 01 06:10:19.566926 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:19.566901 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-lsfzp" event={"ID":"1468808d-a757-4433-8557-2458e1a6ed03","Type":"ContainerStarted","Data":"40beecc2fff81c28ed34525155efbcbe20ebefd93607f3f60710424c5bf4baf3"} Nov 01 06:10:19.567655 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:19.567637 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-lsfzp" Nov 01 06:10:19.569311 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:19.569207 4385 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-lsfzp container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused" start-of-body= Nov 01 06:10:19.569311 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:19.569257 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-lsfzp" podUID="1468808d-a757-4433-8557-2458e1a6ed03" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused" Nov 01 06:10:19.656756 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:19.656719 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:19.836736 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:19.836666 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" podStartSLOduration=2.07865448 podStartE2EDuration="6.836647572s" totalImagesPullingTime="4.757993092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:19.785491799 +0000 UTC m=+29.549291668" watchObservedRunningTime="2026-11-01 06:10:19.836647572 +0000 UTC m=+29.600447429" Nov 01 06:10:19.901686 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:19.901628 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-lsfzp" podStartSLOduration=1.690188918 podStartE2EDuration="18.901611803s" totalImagesPullingTime="17.211422885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:10:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:19.900172411 +0000 UTC m=+29.663972270" watchObservedRunningTime="2026-11-01 06:10:19.901611803 +0000 UTC m=+29.665411659" Nov 01 06:10:19.940082 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:19.940049 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:20.584246 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:20.584217 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-lsfzp" Nov 01 06:10:23.490438 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:23.490403 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54c856867d-jswct" Nov 01 06:10:23.795570 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:23.795491 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:23.941427 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:23.941388 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:24.092001 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:24.091963 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:24.235092 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:24.235056 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:25.294571 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:25.294541 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vvpd2" Nov 01 06:10:25.578857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:25.578807 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-56b4f84ff8-qfxnb" Nov 01 06:10:28.964149 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:28.964112 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:33.174998 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:33.174920 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:33.324922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:33.324885 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:34.227691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:34.227655 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:34.399021 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:34.398991 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:34.640241 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:34.640199 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:34.777454 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:34.777418 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:36.777817 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:10:36.777781 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:10:38.314879 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:38.314827 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:42.973238 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:42.973197 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:43.162749 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:43.162716 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:55.493309 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:55.493272 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:55.641547 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:55.641512 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:57.983033 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:57.982994 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:10:58.120497 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:10:58.120454 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:11:06.778466 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:11:06.778421 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:11:09.688057 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:11:09.688023 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:11:09.878570 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:11:09.878523 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:11:26.170148 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:11:26.170104 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:11:26.321896 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:11:26.321863 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:11:31.483068 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:11:31.483028 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:11:31.672589 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:11:31.672557 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:11:31.863878 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:11:31.863829 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:11:32.005677 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:11:32.005648 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:11:34.578747 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:11:34.578708 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:11:36.778594 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:11:36.778552 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:11:42.985281 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:11:42.985241 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:11:43.135818 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:11:43.135784 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:11:43.286064 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:11:43.285970 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:11:43.799568 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:11:43.799534 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:11:43.995234 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:11:43.995202 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:12:06.778672 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:12:06.778593 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:12:27.673583 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:12:27.673543 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:12:34.908818 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:12:34.908778 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:12:36.778592 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:12:36.778541 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:13:06.778523 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:13:06.778485 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:13:12.154252 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:12.154213 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:13:18.813798 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:18.813765 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:13:19.084231 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:19.084190 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:13:19.619675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:19.619634 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc"] Nov 01 06:13:19.621986 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:19.621961 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc" Nov 01 06:13:19.674653 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:19.674615 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 01 06:13:19.705756 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:19.705708 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8gb4\" (UniqueName: \"kubernetes.io/projected/05ab5c46-8257-4e49-b936-c83eb203b08f-kube-api-access-x8gb4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc\" (UID: \"05ab5c46-8257-4e49-b936-c83eb203b08f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc" Nov 01 06:13:19.705756 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:19.705752 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05ab5c46-8257-4e49-b936-c83eb203b08f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc\" (UID: \"05ab5c46-8257-4e49-b936-c83eb203b08f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc" Nov 01 06:13:19.706093 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:19.705856 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05ab5c46-8257-4e49-b936-c83eb203b08f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc\" (UID: \"05ab5c46-8257-4e49-b936-c83eb203b08f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc" Nov 01 06:13:19.724689 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:19.724659 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-crw5c\"" Nov 01 06:13:19.724689 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:19.724659 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 01 06:13:19.725825 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:19.725807 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc"] Nov 01 06:13:19.807081 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:19.807039 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8gb4\" (UniqueName: \"kubernetes.io/projected/05ab5c46-8257-4e49-b936-c83eb203b08f-kube-api-access-x8gb4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc\" (UID: \"05ab5c46-8257-4e49-b936-c83eb203b08f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc" Nov 01 06:13:19.807081 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:19.807082 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05ab5c46-8257-4e49-b936-c83eb203b08f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc\" (UID: \"05ab5c46-8257-4e49-b936-c83eb203b08f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc" Nov 01 06:13:19.807293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:19.807111 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05ab5c46-8257-4e49-b936-c83eb203b08f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc\" (UID: \"05ab5c46-8257-4e49-b936-c83eb203b08f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc" Nov 01 06:13:19.807434 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:19.807419 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05ab5c46-8257-4e49-b936-c83eb203b08f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc\" (UID: \"05ab5c46-8257-4e49-b936-c83eb203b08f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc" Nov 01 06:13:19.807497 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:19.807476 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05ab5c46-8257-4e49-b936-c83eb203b08f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc\" (UID: \"05ab5c46-8257-4e49-b936-c83eb203b08f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc" Nov 01 06:13:19.864656 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:19.864623 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8gb4\" (UniqueName: \"kubernetes.io/projected/05ab5c46-8257-4e49-b936-c83eb203b08f-kube-api-access-x8gb4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc\" (UID: \"05ab5c46-8257-4e49-b936-c83eb203b08f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc" Nov 01 06:13:19.933497 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:19.933411 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc" Nov 01 06:13:20.144328 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.144297 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:13:20.305285 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.305206 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc"] Nov 01 06:13:20.308339 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:13:20.308309 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ab5c46_8257_4e49_b936_c83eb203b08f.slice/crio-06bdfd9afd40ee08fc85ed43a96b861711a550f2cdc41c1ff19a6e2badb6d420 WatchSource:0}: Error finding container 06bdfd9afd40ee08fc85ed43a96b861711a550f2cdc41c1ff19a6e2badb6d420: Status 404 returned error can't find the container with id 06bdfd9afd40ee08fc85ed43a96b861711a550f2cdc41c1ff19a6e2badb6d420 Nov 01 06:13:20.356353 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.356317 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:13:20.726461 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.726426 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh"] Nov 01 06:13:20.728955 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.728939 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh" Nov 01 06:13:20.783616 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.783554 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-9vpsv\"" Nov 01 06:13:20.785373 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.785347 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh"] Nov 01 06:13:20.815207 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.815177 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77c51a13-98cf-45ab-a049-bac31d7cada0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh\" (UID: \"77c51a13-98cf-45ab-a049-bac31d7cada0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh" Nov 01 06:13:20.815378 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.815233 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnc8q\" (UniqueName: \"kubernetes.io/projected/77c51a13-98cf-45ab-a049-bac31d7cada0-kube-api-access-lnc8q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh\" (UID: \"77c51a13-98cf-45ab-a049-bac31d7cada0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh" Nov 01 06:13:20.815378 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.815261 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77c51a13-98cf-45ab-a049-bac31d7cada0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh\" (UID: \"77c51a13-98cf-45ab-a049-bac31d7cada0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh" Nov 01 06:13:20.825358 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.825324 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x"] Nov 01 06:13:20.827421 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.827401 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x" Nov 01 06:13:20.832868 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.832824 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9"] Nov 01 06:13:20.835241 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.835220 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9" Nov 01 06:13:20.879059 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.879027 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-sw4xg\"" Nov 01 06:13:20.895037 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.895005 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x"] Nov 01 06:13:20.898736 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.898706 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9"] Nov 01 06:13:20.908907 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.908880 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-4hbwn\"" Nov 01 06:13:20.916198 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.916168 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnc8q\" (UniqueName: \"kubernetes.io/projected/77c51a13-98cf-45ab-a049-bac31d7cada0-kube-api-access-lnc8q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh\" (UID: \"77c51a13-98cf-45ab-a049-bac31d7cada0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh" Nov 01 06:13:20.916362 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.916224 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9x7f\" (UniqueName: \"kubernetes.io/projected/e49365a6-eff4-4d06-a968-9a05e25a70c0-kube-api-access-f9x7f\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9\" (UID: \"e49365a6-eff4-4d06-a968-9a05e25a70c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9" Nov 01 06:13:20.916362 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.916252 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77c51a13-98cf-45ab-a049-bac31d7cada0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh\" (UID: \"77c51a13-98cf-45ab-a049-bac31d7cada0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh" Nov 01 06:13:20.916362 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.916283 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9306df59-d376-411e-8fa4-6606ed054782-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x\" (UID: \"9306df59-d376-411e-8fa4-6606ed054782\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x" Nov 01 06:13:20.916362 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.916320 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e49365a6-eff4-4d06-a968-9a05e25a70c0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9\" (UID: \"e49365a6-eff4-4d06-a968-9a05e25a70c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9" Nov 01 06:13:20.916516 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.916387 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77c51a13-98cf-45ab-a049-bac31d7cada0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh\" (UID: \"77c51a13-98cf-45ab-a049-bac31d7cada0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh" Nov 01 06:13:20.916516 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.916471 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9306df59-d376-411e-8fa4-6606ed054782-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x\" (UID: \"9306df59-d376-411e-8fa4-6606ed054782\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x" Nov 01 06:13:20.916516 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.916503 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cclxx\" (UniqueName: \"kubernetes.io/projected/9306df59-d376-411e-8fa4-6606ed054782-kube-api-access-cclxx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x\" (UID: \"9306df59-d376-411e-8fa4-6606ed054782\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x" Nov 01 06:13:20.916618 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.916547 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e49365a6-eff4-4d06-a968-9a05e25a70c0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9\" (UID: \"e49365a6-eff4-4d06-a968-9a05e25a70c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9" Nov 01 06:13:20.916734 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.916715 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77c51a13-98cf-45ab-a049-bac31d7cada0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh\" (UID: \"77c51a13-98cf-45ab-a049-bac31d7cada0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh" Nov 01 06:13:20.916734 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.916726 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77c51a13-98cf-45ab-a049-bac31d7cada0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh\" (UID: \"77c51a13-98cf-45ab-a049-bac31d7cada0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh" Nov 01 06:13:20.926339 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.926308 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv"] Nov 01 06:13:20.928399 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.928385 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv" Nov 01 06:13:20.972705 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:20.972673 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnc8q\" (UniqueName: \"kubernetes.io/projected/77c51a13-98cf-45ab-a049-bac31d7cada0-kube-api-access-lnc8q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh\" (UID: \"77c51a13-98cf-45ab-a049-bac31d7cada0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh" Nov 01 06:13:21.008890 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.008796 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-6fzqk\"" Nov 01 06:13:21.017100 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.017069 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e49365a6-eff4-4d06-a968-9a05e25a70c0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9\" (UID: \"e49365a6-eff4-4d06-a968-9a05e25a70c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9" Nov 01 06:13:21.017240 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.017112 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9306df59-d376-411e-8fa4-6606ed054782-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x\" (UID: \"9306df59-d376-411e-8fa4-6606ed054782\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x" Nov 01 06:13:21.017240 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.017132 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cclxx\" (UniqueName: \"kubernetes.io/projected/9306df59-d376-411e-8fa4-6606ed054782-kube-api-access-cclxx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x\" (UID: \"9306df59-d376-411e-8fa4-6606ed054782\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x" Nov 01 06:13:21.017240 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.017175 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c591503c-5b74-45c7-a3f5-13cbecb15650-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv\" (UID: \"c591503c-5b74-45c7-a3f5-13cbecb15650\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv" Nov 01 06:13:21.017240 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.017207 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e49365a6-eff4-4d06-a968-9a05e25a70c0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9\" (UID: \"e49365a6-eff4-4d06-a968-9a05e25a70c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9" Nov 01 06:13:21.017413 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.017240 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjlfv\" (UniqueName: \"kubernetes.io/projected/c591503c-5b74-45c7-a3f5-13cbecb15650-kube-api-access-mjlfv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv\" (UID: \"c591503c-5b74-45c7-a3f5-13cbecb15650\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv" Nov 01 06:13:21.022095 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.017749 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9306df59-d376-411e-8fa4-6606ed054782-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x\" (UID: \"9306df59-d376-411e-8fa4-6606ed054782\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x" Nov 01 06:13:21.022095 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.017789 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e49365a6-eff4-4d06-a968-9a05e25a70c0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9\" (UID: \"e49365a6-eff4-4d06-a968-9a05e25a70c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9" Nov 01 06:13:21.022095 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.017807 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9x7f\" (UniqueName: \"kubernetes.io/projected/e49365a6-eff4-4d06-a968-9a05e25a70c0-kube-api-access-f9x7f\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9\" (UID: \"e49365a6-eff4-4d06-a968-9a05e25a70c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9" Nov 01 06:13:21.022095 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.017907 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9306df59-d376-411e-8fa4-6606ed054782-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x\" (UID: \"9306df59-d376-411e-8fa4-6606ed054782\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x" Nov 01 06:13:21.022095 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.017949 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c591503c-5b74-45c7-a3f5-13cbecb15650-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv\" (UID: \"c591503c-5b74-45c7-a3f5-13cbecb15650\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv" Nov 01 06:13:21.022095 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.018072 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e49365a6-eff4-4d06-a968-9a05e25a70c0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9\" (UID: \"e49365a6-eff4-4d06-a968-9a05e25a70c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9" Nov 01 06:13:21.022095 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.018271 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9306df59-d376-411e-8fa4-6606ed054782-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x\" (UID: \"9306df59-d376-411e-8fa4-6606ed054782\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x" Nov 01 06:13:21.028977 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.028944 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv"] Nov 01 06:13:21.034500 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.034471 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm"] Nov 01 06:13:21.036725 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.036707 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm" Nov 01 06:13:21.040677 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.040659 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh" Nov 01 06:13:21.074507 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.074467 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cclxx\" (UniqueName: \"kubernetes.io/projected/9306df59-d376-411e-8fa4-6606ed054782-kube-api-access-cclxx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x\" (UID: \"9306df59-d376-411e-8fa4-6606ed054782\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x" Nov 01 06:13:21.074996 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.074975 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9x7f\" (UniqueName: \"kubernetes.io/projected/e49365a6-eff4-4d06-a968-9a05e25a70c0-kube-api-access-f9x7f\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9\" (UID: \"e49365a6-eff4-4d06-a968-9a05e25a70c0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9" Nov 01 06:13:21.091510 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.091390 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-lc4cn\"" Nov 01 06:13:21.093786 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.093759 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm"] Nov 01 06:13:21.112401 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.112367 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:13:21.119237 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.119198 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c591503c-5b74-45c7-a3f5-13cbecb15650-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv\" (UID: \"c591503c-5b74-45c7-a3f5-13cbecb15650\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv" Nov 01 06:13:21.119389 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.119244 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjlfv\" (UniqueName: \"kubernetes.io/projected/c591503c-5b74-45c7-a3f5-13cbecb15650-kube-api-access-mjlfv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv\" (UID: \"c591503c-5b74-45c7-a3f5-13cbecb15650\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv" Nov 01 06:13:21.119389 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.119286 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c591503c-5b74-45c7-a3f5-13cbecb15650-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv\" (UID: \"c591503c-5b74-45c7-a3f5-13cbecb15650\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv" Nov 01 06:13:21.119389 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.119319 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpv4g\" (UniqueName: \"kubernetes.io/projected/9b1e735a-5598-41d3-b557-5161b194e7d3-kube-api-access-jpv4g\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm\" (UID: \"9b1e735a-5598-41d3-b557-5161b194e7d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm" Nov 01 06:13:21.119389 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.119354 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b1e735a-5598-41d3-b557-5161b194e7d3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm\" (UID: \"9b1e735a-5598-41d3-b557-5161b194e7d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm" Nov 01 06:13:21.119389 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.119385 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b1e735a-5598-41d3-b557-5161b194e7d3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm\" (UID: \"9b1e735a-5598-41d3-b557-5161b194e7d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm" Nov 01 06:13:21.119602 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.119593 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c591503c-5b74-45c7-a3f5-13cbecb15650-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv\" (UID: \"c591503c-5b74-45c7-a3f5-13cbecb15650\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv" Nov 01 06:13:21.120419 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.120398 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c591503c-5b74-45c7-a3f5-13cbecb15650-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv\" (UID: \"c591503c-5b74-45c7-a3f5-13cbecb15650\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv" Nov 01 06:13:21.139725 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.139689 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x" Nov 01 06:13:21.146709 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.146686 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9" Nov 01 06:13:21.165785 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.165750 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc" event={"ID":"05ab5c46-8257-4e49-b936-c83eb203b08f","Type":"ContainerStarted","Data":"06bdfd9afd40ee08fc85ed43a96b861711a550f2cdc41c1ff19a6e2badb6d420"} Nov 01 06:13:21.176893 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.176849 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjlfv\" (UniqueName: \"kubernetes.io/projected/c591503c-5b74-45c7-a3f5-13cbecb15650-kube-api-access-mjlfv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv\" (UID: \"c591503c-5b74-45c7-a3f5-13cbecb15650\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv" Nov 01 06:13:21.220919 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.220878 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpv4g\" (UniqueName: \"kubernetes.io/projected/9b1e735a-5598-41d3-b557-5161b194e7d3-kube-api-access-jpv4g\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm\" (UID: \"9b1e735a-5598-41d3-b557-5161b194e7d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm" Nov 01 06:13:21.221092 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.220954 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b1e735a-5598-41d3-b557-5161b194e7d3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm\" (UID: \"9b1e735a-5598-41d3-b557-5161b194e7d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm" Nov 01 06:13:21.221092 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.220996 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b1e735a-5598-41d3-b557-5161b194e7d3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm\" (UID: \"9b1e735a-5598-41d3-b557-5161b194e7d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm" Nov 01 06:13:21.221387 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.221354 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b1e735a-5598-41d3-b557-5161b194e7d3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm\" (UID: \"9b1e735a-5598-41d3-b557-5161b194e7d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm" Nov 01 06:13:21.221468 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.221381 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b1e735a-5598-41d3-b557-5161b194e7d3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm\" (UID: \"9b1e735a-5598-41d3-b557-5161b194e7d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm" Nov 01 06:13:21.240197 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.240169 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv" Nov 01 06:13:21.278063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.277992 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpv4g\" (UniqueName: \"kubernetes.io/projected/9b1e735a-5598-41d3-b557-5161b194e7d3-kube-api-access-jpv4g\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm\" (UID: \"9b1e735a-5598-41d3-b557-5161b194e7d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm" Nov 01 06:13:21.349177 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.349139 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm" Nov 01 06:13:21.414690 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.414655 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh"] Nov 01 06:13:21.417803 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:13:21.417776 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c51a13_98cf_45ab_a049_bac31d7cada0.slice/crio-c93d144105f00585ce4a719724caf0eae7fd1505899e6f67b24320e957d65eb0 WatchSource:0}: Error finding container c93d144105f00585ce4a719724caf0eae7fd1505899e6f67b24320e957d65eb0: Status 404 returned error can't find the container with id c93d144105f00585ce4a719724caf0eae7fd1505899e6f67b24320e957d65eb0 Nov 01 06:13:21.515236 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.515212 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x"] Nov 01 06:13:21.517400 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:13:21.517371 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9306df59_d376_411e_8fa4_6606ed054782.slice/crio-295abd59bdadadf7018dd006d8799b2fec39c31bde6f7188549773a5bb0ce55a WatchSource:0}: Error finding container 295abd59bdadadf7018dd006d8799b2fec39c31bde6f7188549773a5bb0ce55a: Status 404 returned error can't find the container with id 295abd59bdadadf7018dd006d8799b2fec39c31bde6f7188549773a5bb0ce55a Nov 01 06:13:21.524679 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.524640 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9"] Nov 01 06:13:21.527912 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:13:21.527886 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49365a6_eff4_4d06_a968_9a05e25a70c0.slice/crio-ad18db0979704ffb5002ccf0f4cd6822ea23e1c3e244a1406974556376e32b41 WatchSource:0}: Error finding container ad18db0979704ffb5002ccf0f4cd6822ea23e1c3e244a1406974556376e32b41: Status 404 returned error can't find the container with id ad18db0979704ffb5002ccf0f4cd6822ea23e1c3e244a1406974556376e32b41 Nov 01 06:13:21.617188 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.617157 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv"] Nov 01 06:13:21.620941 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:13:21.620915 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc591503c_5b74_45c7_a3f5_13cbecb15650.slice/crio-763b74bccf2609173bc2c029c2631481a2a6daec7d6af67f9432489fef777ac1 WatchSource:0}: Error finding container 763b74bccf2609173bc2c029c2631481a2a6daec7d6af67f9432489fef777ac1: Status 404 returned error can't find the container with id 763b74bccf2609173bc2c029c2631481a2a6daec7d6af67f9432489fef777ac1 Nov 01 06:13:21.713728 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:21.713697 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm"] Nov 01 06:13:21.717506 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:13:21.717471 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1e735a_5598_41d3_b557_5161b194e7d3.slice/crio-17186c25dd41ea2fd7bfba1abb2f98fea82b482c52095ac381c62ea2e6039a27 WatchSource:0}: Error finding container 17186c25dd41ea2fd7bfba1abb2f98fea82b482c52095ac381c62ea2e6039a27: Status 404 returned error can't find the container with id 17186c25dd41ea2fd7bfba1abb2f98fea82b482c52095ac381c62ea2e6039a27 Nov 01 06:13:22.170711 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:22.170671 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv" event={"ID":"c591503c-5b74-45c7-a3f5-13cbecb15650","Type":"ContainerStarted","Data":"763b74bccf2609173bc2c029c2631481a2a6daec7d6af67f9432489fef777ac1"} Nov 01 06:13:22.171889 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:22.171863 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9" event={"ID":"e49365a6-eff4-4d06-a968-9a05e25a70c0","Type":"ContainerStarted","Data":"ad18db0979704ffb5002ccf0f4cd6822ea23e1c3e244a1406974556376e32b41"} Nov 01 06:13:22.173042 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:22.173015 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x" event={"ID":"9306df59-d376-411e-8fa4-6606ed054782","Type":"ContainerStarted","Data":"295abd59bdadadf7018dd006d8799b2fec39c31bde6f7188549773a5bb0ce55a"} Nov 01 06:13:22.174207 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:22.174187 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm" event={"ID":"9b1e735a-5598-41d3-b557-5161b194e7d3","Type":"ContainerStarted","Data":"17186c25dd41ea2fd7bfba1abb2f98fea82b482c52095ac381c62ea2e6039a27"} Nov 01 06:13:22.175263 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:22.175243 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh" event={"ID":"77c51a13-98cf-45ab-a049-bac31d7cada0","Type":"ContainerStarted","Data":"c93d144105f00585ce4a719724caf0eae7fd1505899e6f67b24320e957d65eb0"} Nov 01 06:13:22.880120 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:22.880084 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl"] Nov 01 06:13:22.883007 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:22.882982 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl" Nov 01 06:13:22.936795 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:22.936647 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvt6c\" (UniqueName: \"kubernetes.io/projected/2ea4d43f-2d68-435a-adec-081098268d28-kube-api-access-jvt6c\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl\" (UID: \"2ea4d43f-2d68-435a-adec-081098268d28\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl" Nov 01 06:13:22.936795 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:22.936690 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ea4d43f-2d68-435a-adec-081098268d28-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl\" (UID: \"2ea4d43f-2d68-435a-adec-081098268d28\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl" Nov 01 06:13:22.936795 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:22.936709 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ea4d43f-2d68-435a-adec-081098268d28-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl\" (UID: \"2ea4d43f-2d68-435a-adec-081098268d28\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl" Nov 01 06:13:22.987783 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:22.987748 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-h5zh9\"" Nov 01 06:13:22.990285 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:22.990255 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl"] Nov 01 06:13:23.038198 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:23.038156 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvt6c\" (UniqueName: \"kubernetes.io/projected/2ea4d43f-2d68-435a-adec-081098268d28-kube-api-access-jvt6c\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl\" (UID: \"2ea4d43f-2d68-435a-adec-081098268d28\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl" Nov 01 06:13:23.038396 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:23.038220 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ea4d43f-2d68-435a-adec-081098268d28-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl\" (UID: \"2ea4d43f-2d68-435a-adec-081098268d28\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl" Nov 01 06:13:23.038396 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:23.038253 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ea4d43f-2d68-435a-adec-081098268d28-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl\" (UID: \"2ea4d43f-2d68-435a-adec-081098268d28\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl" Nov 01 06:13:23.038733 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:23.038708 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ea4d43f-2d68-435a-adec-081098268d28-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl\" (UID: \"2ea4d43f-2d68-435a-adec-081098268d28\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl" Nov 01 06:13:23.038812 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:23.038737 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ea4d43f-2d68-435a-adec-081098268d28-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl\" (UID: \"2ea4d43f-2d68-435a-adec-081098268d28\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl" Nov 01 06:13:23.110503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:23.110467 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvt6c\" (UniqueName: \"kubernetes.io/projected/2ea4d43f-2d68-435a-adec-081098268d28-kube-api-access-jvt6c\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl\" (UID: \"2ea4d43f-2d68-435a-adec-081098268d28\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl" Nov 01 06:13:23.201976 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:23.201635 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl" Nov 01 06:13:23.627901 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:23.627786 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl"] Nov 01 06:13:23.634765 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:13:23.634646 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea4d43f_2d68_435a_adec_081098268d28.slice/crio-9259989ada70648360eb7ff0a948df3a87958647bb3cc111d6dac6c3eb585938 WatchSource:0}: Error finding container 9259989ada70648360eb7ff0a948df3a87958647bb3cc111d6dac6c3eb585938: Status 404 returned error can't find the container with id 9259989ada70648360eb7ff0a948df3a87958647bb3cc111d6dac6c3eb585938 Nov 01 06:13:24.193679 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:24.193640 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl" event={"ID":"2ea4d43f-2d68-435a-adec-081098268d28","Type":"ContainerStarted","Data":"9259989ada70648360eb7ff0a948df3a87958647bb3cc111d6dac6c3eb585938"} Nov 01 06:13:25.174947 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:25.173151 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j"] Nov 01 06:13:25.176698 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:25.176672 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j" Nov 01 06:13:25.260606 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:25.260334 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/885a807a-2d27-4f2b-8a04-7a6c1388820b-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j\" (UID: \"885a807a-2d27-4f2b-8a04-7a6c1388820b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j" Nov 01 06:13:25.260606 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:25.260397 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/885a807a-2d27-4f2b-8a04-7a6c1388820b-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j\" (UID: \"885a807a-2d27-4f2b-8a04-7a6c1388820b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j" Nov 01 06:13:25.260606 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:25.260435 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mbkt\" (UniqueName: \"kubernetes.io/projected/885a807a-2d27-4f2b-8a04-7a6c1388820b-kube-api-access-4mbkt\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j\" (UID: \"885a807a-2d27-4f2b-8a04-7a6c1388820b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j" Nov 01 06:13:25.283159 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:25.282995 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j"] Nov 01 06:13:25.362074 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:25.362036 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/885a807a-2d27-4f2b-8a04-7a6c1388820b-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j\" (UID: \"885a807a-2d27-4f2b-8a04-7a6c1388820b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j" Nov 01 06:13:25.362264 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:25.362109 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mbkt\" (UniqueName: \"kubernetes.io/projected/885a807a-2d27-4f2b-8a04-7a6c1388820b-kube-api-access-4mbkt\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j\" (UID: \"885a807a-2d27-4f2b-8a04-7a6c1388820b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j" Nov 01 06:13:25.362264 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:25.362209 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/885a807a-2d27-4f2b-8a04-7a6c1388820b-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j\" (UID: \"885a807a-2d27-4f2b-8a04-7a6c1388820b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j" Nov 01 06:13:25.362726 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:25.362699 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/885a807a-2d27-4f2b-8a04-7a6c1388820b-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j\" (UID: \"885a807a-2d27-4f2b-8a04-7a6c1388820b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j" Nov 01 06:13:25.363090 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:25.363039 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/885a807a-2d27-4f2b-8a04-7a6c1388820b-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j\" (UID: \"885a807a-2d27-4f2b-8a04-7a6c1388820b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j" Nov 01 06:13:25.439234 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:25.439163 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mbkt\" (UniqueName: \"kubernetes.io/projected/885a807a-2d27-4f2b-8a04-7a6c1388820b-kube-api-access-4mbkt\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j\" (UID: \"885a807a-2d27-4f2b-8a04-7a6c1388820b\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j" Nov 01 06:13:25.496869 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:25.496779 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j" Nov 01 06:13:25.905331 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:25.905302 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j"] Nov 01 06:13:25.910076 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:13:25.910041 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod885a807a_2d27_4f2b_8a04_7a6c1388820b.slice/crio-1eae191da64eeb6a31e8357cbe2bb98c22c1e39c8fbb83882ad1f2d6ff2b49e5 WatchSource:0}: Error finding container 1eae191da64eeb6a31e8357cbe2bb98c22c1e39c8fbb83882ad1f2d6ff2b49e5: Status 404 returned error can't find the container with id 1eae191da64eeb6a31e8357cbe2bb98c22c1e39c8fbb83882ad1f2d6ff2b49e5 Nov 01 06:13:26.217790 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:26.217698 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j" event={"ID":"885a807a-2d27-4f2b-8a04-7a6c1388820b","Type":"ContainerStarted","Data":"1eae191da64eeb6a31e8357cbe2bb98c22c1e39c8fbb83882ad1f2d6ff2b49e5"} Nov 01 06:13:26.766586 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:26.766550 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:13:26.926782 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:26.926748 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:13:27.159359 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.159325 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd"] Nov 01 06:13:27.162418 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.162386 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.233400 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.232631 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-g6wqc\"" Nov 01 06:13:27.233400 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.233303 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 06:13:27.233400 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.233367 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 06:13:27.269554 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.267132 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:13:27.277000 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.274937 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd"] Nov 01 06:13:27.277000 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.275065 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 06:13:27.279979 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.279682 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.279979 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.279725 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.279979 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.279771 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.279979 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.279807 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.279979 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.279845 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.279979 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.279872 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.279979 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.279904 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.279979 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.279958 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.280480 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.279988 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mh65\" (UniqueName: \"kubernetes.io/projected/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-kube-api-access-5mh65\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.381241 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.381202 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.381415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.381262 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.381415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.381295 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.381415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.381322 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.381415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.381357 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.381661 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.381416 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.381661 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.381449 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mh65\" (UniqueName: \"kubernetes.io/projected/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-kube-api-access-5mh65\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.381661 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.381506 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.381661 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.381538 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.382195 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.382019 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.382360 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.382287 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.383676 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.383607 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.384100 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.383781 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.384100 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.383877 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.392438 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.392362 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.397480 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.397408 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.453985 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.453865 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.454146 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.454096 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mh65\" (UniqueName: \"kubernetes.io/projected/bf4f43cd-9da8-4bd6-9b49-9c7329b5480a-kube-api-access-5mh65\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8ckbd\" (UID: \"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.454402 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.454378 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7"] Nov 01 06:13:27.457285 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.457262 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.479864 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.479614 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:27.517627 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.517508 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7"] Nov 01 06:13:27.543649 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.543545 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-6lm64\"" Nov 01 06:13:27.571300 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.571240 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:13:27.584653 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.584005 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28f52bd7-a41a-4af8-9f66-911b3fe24b58-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.584653 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.584062 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28f52bd7-a41a-4af8-9f66-911b3fe24b58-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.584653 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.584098 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28f52bd7-a41a-4af8-9f66-911b3fe24b58-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.584653 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.584125 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zmw8\" (UniqueName: \"kubernetes.io/projected/28f52bd7-a41a-4af8-9f66-911b3fe24b58-kube-api-access-5zmw8\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.584653 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.584193 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f52bd7-a41a-4af8-9f66-911b3fe24b58-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.584653 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.584217 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28f52bd7-a41a-4af8-9f66-911b3fe24b58-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.584653 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.584264 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f52bd7-a41a-4af8-9f66-911b3fe24b58-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.584653 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.584302 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28f52bd7-a41a-4af8-9f66-911b3fe24b58-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.584653 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.584326 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28f52bd7-a41a-4af8-9f66-911b3fe24b58-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.686680 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.685758 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28f52bd7-a41a-4af8-9f66-911b3fe24b58-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.686680 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.685799 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28f52bd7-a41a-4af8-9f66-911b3fe24b58-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.686680 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.685824 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28f52bd7-a41a-4af8-9f66-911b3fe24b58-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.686680 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.685854 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zmw8\" (UniqueName: \"kubernetes.io/projected/28f52bd7-a41a-4af8-9f66-911b3fe24b58-kube-api-access-5zmw8\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.686680 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.685907 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f52bd7-a41a-4af8-9f66-911b3fe24b58-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.686680 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.685922 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28f52bd7-a41a-4af8-9f66-911b3fe24b58-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.686680 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.685951 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f52bd7-a41a-4af8-9f66-911b3fe24b58-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.686680 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.685988 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28f52bd7-a41a-4af8-9f66-911b3fe24b58-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.686680 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.686010 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28f52bd7-a41a-4af8-9f66-911b3fe24b58-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.688424 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.687871 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28f52bd7-a41a-4af8-9f66-911b3fe24b58-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.688424 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.688247 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f52bd7-a41a-4af8-9f66-911b3fe24b58-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.688603 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.688433 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28f52bd7-a41a-4af8-9f66-911b3fe24b58-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.688603 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.688545 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28f52bd7-a41a-4af8-9f66-911b3fe24b58-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.688804 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.688740 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28f52bd7-a41a-4af8-9f66-911b3fe24b58-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.698521 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.698451 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28f52bd7-a41a-4af8-9f66-911b3fe24b58-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.704609 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.704542 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28f52bd7-a41a-4af8-9f66-911b3fe24b58-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.710935 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.710861 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:13:27.752130 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.752102 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zmw8\" (UniqueName: \"kubernetes.io/projected/28f52bd7-a41a-4af8-9f66-911b3fe24b58-kube-api-access-5zmw8\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.759242 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.759169 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28f52bd7-a41a-4af8-9f66-911b3fe24b58-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7\" (UID: \"28f52bd7-a41a-4af8-9f66-911b3fe24b58\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.783051 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.783014 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:27.797132 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.795029 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv"] Nov 01 06:13:27.798102 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.798080 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv" Nov 01 06:13:27.815413 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.815384 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc"] Nov 01 06:13:27.818812 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.818787 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc" Nov 01 06:13:27.855604 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.855574 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7"] Nov 01 06:13:27.870576 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.870514 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv"] Nov 01 06:13:27.870794 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.870650 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7" Nov 01 06:13:27.877170 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.877135 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl"] Nov 01 06:13:27.881149 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.881125 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl" Nov 01 06:13:27.885564 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.885540 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc"] Nov 01 06:13:27.889667 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.888990 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nppqk\" (UniqueName: \"kubernetes.io/projected/75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8-kube-api-access-nppqk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv\" (UID: \"75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv" Nov 01 06:13:27.889667 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.889040 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e102dd21-a348-4c82-8cc1-3938b7e68387-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc\" (UID: \"e102dd21-a348-4c82-8cc1-3938b7e68387\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc" Nov 01 06:13:27.889667 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.889135 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e102dd21-a348-4c82-8cc1-3938b7e68387-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc\" (UID: \"e102dd21-a348-4c82-8cc1-3938b7e68387\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc" Nov 01 06:13:27.889667 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.889160 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv\" (UID: \"75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv" Nov 01 06:13:27.889667 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.889210 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv\" (UID: \"75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv" Nov 01 06:13:27.889667 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.889253 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdlp7\" (UniqueName: \"kubernetes.io/projected/e102dd21-a348-4c82-8cc1-3938b7e68387-kube-api-access-vdlp7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc\" (UID: \"e102dd21-a348-4c82-8cc1-3938b7e68387\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc" Nov 01 06:13:27.909177 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.909151 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7"] Nov 01 06:13:27.937238 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.937203 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl"] Nov 01 06:13:27.965433 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.965309 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd"] Nov 01 06:13:27.974343 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:13:27.974150 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf4f43cd_9da8_4bd6_9b49_9c7329b5480a.slice/crio-57e7b59a3a11095784e4703354c530af84fec923fc60b6404fc1e8171e876225 WatchSource:0}: Error finding container 57e7b59a3a11095784e4703354c530af84fec923fc60b6404fc1e8171e876225: Status 404 returned error can't find the container with id 57e7b59a3a11095784e4703354c530af84fec923fc60b6404fc1e8171e876225 Nov 01 06:13:27.992542 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.990117 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nppqk\" (UniqueName: \"kubernetes.io/projected/75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8-kube-api-access-nppqk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv\" (UID: \"75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv" Nov 01 06:13:27.992542 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.990160 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e102dd21-a348-4c82-8cc1-3938b7e68387-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc\" (UID: \"e102dd21-a348-4c82-8cc1-3938b7e68387\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc" Nov 01 06:13:27.992542 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.990928 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d2abea4-3ca9-4866-831d-644af9c391b1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7\" (UID: \"3d2abea4-3ca9-4866-831d-644af9c391b1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7" Nov 01 06:13:27.992542 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.990982 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b02fadf0-b23a-4006-a0d8-7669d9565d7b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl\" (UID: \"b02fadf0-b23a-4006-a0d8-7669d9565d7b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl" Nov 01 06:13:27.992542 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.991028 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d2abea4-3ca9-4866-831d-644af9c391b1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7\" (UID: \"3d2abea4-3ca9-4866-831d-644af9c391b1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7" Nov 01 06:13:27.992542 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.991074 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e102dd21-a348-4c82-8cc1-3938b7e68387-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc\" (UID: \"e102dd21-a348-4c82-8cc1-3938b7e68387\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc" Nov 01 06:13:27.992542 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.991129 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv\" (UID: \"75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv" Nov 01 06:13:27.992542 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.991167 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9fxv\" (UniqueName: \"kubernetes.io/projected/b02fadf0-b23a-4006-a0d8-7669d9565d7b-kube-api-access-s9fxv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl\" (UID: \"b02fadf0-b23a-4006-a0d8-7669d9565d7b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl" Nov 01 06:13:27.992542 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.991202 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b02fadf0-b23a-4006-a0d8-7669d9565d7b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl\" (UID: \"b02fadf0-b23a-4006-a0d8-7669d9565d7b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl" Nov 01 06:13:27.992542 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.991245 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv\" (UID: \"75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv" Nov 01 06:13:27.992542 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.991295 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdlp7\" (UniqueName: \"kubernetes.io/projected/e102dd21-a348-4c82-8cc1-3938b7e68387-kube-api-access-vdlp7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc\" (UID: \"e102dd21-a348-4c82-8cc1-3938b7e68387\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc" Nov 01 06:13:27.992542 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.991329 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdntw\" (UniqueName: \"kubernetes.io/projected/3d2abea4-3ca9-4866-831d-644af9c391b1-kube-api-access-jdntw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7\" (UID: \"3d2abea4-3ca9-4866-831d-644af9c391b1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7" Nov 01 06:13:27.992542 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.991664 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e102dd21-a348-4c82-8cc1-3938b7e68387-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc\" (UID: \"e102dd21-a348-4c82-8cc1-3938b7e68387\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc" Nov 01 06:13:27.992542 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.991986 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv\" (UID: \"75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv" Nov 01 06:13:27.992542 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.992268 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv\" (UID: \"75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv" Nov 01 06:13:27.993391 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:27.992941 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e102dd21-a348-4c82-8cc1-3938b7e68387-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc\" (UID: \"e102dd21-a348-4c82-8cc1-3938b7e68387\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc" Nov 01 06:13:28.048330 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.048268 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nppqk\" (UniqueName: \"kubernetes.io/projected/75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8-kube-api-access-nppqk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv\" (UID: \"75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv" Nov 01 06:13:28.049118 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.049072 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdlp7\" (UniqueName: \"kubernetes.io/projected/e102dd21-a348-4c82-8cc1-3938b7e68387-kube-api-access-vdlp7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc\" (UID: \"e102dd21-a348-4c82-8cc1-3938b7e68387\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc" Nov 01 06:13:28.092869 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.092044 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdntw\" (UniqueName: \"kubernetes.io/projected/3d2abea4-3ca9-4866-831d-644af9c391b1-kube-api-access-jdntw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7\" (UID: \"3d2abea4-3ca9-4866-831d-644af9c391b1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7" Nov 01 06:13:28.092869 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.092115 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d2abea4-3ca9-4866-831d-644af9c391b1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7\" (UID: \"3d2abea4-3ca9-4866-831d-644af9c391b1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7" Nov 01 06:13:28.092869 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.092147 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b02fadf0-b23a-4006-a0d8-7669d9565d7b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl\" (UID: \"b02fadf0-b23a-4006-a0d8-7669d9565d7b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl" Nov 01 06:13:28.092869 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.092191 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d2abea4-3ca9-4866-831d-644af9c391b1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7\" (UID: \"3d2abea4-3ca9-4866-831d-644af9c391b1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7" Nov 01 06:13:28.092869 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.092237 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9fxv\" (UniqueName: \"kubernetes.io/projected/b02fadf0-b23a-4006-a0d8-7669d9565d7b-kube-api-access-s9fxv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl\" (UID: \"b02fadf0-b23a-4006-a0d8-7669d9565d7b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl" Nov 01 06:13:28.092869 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.092265 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b02fadf0-b23a-4006-a0d8-7669d9565d7b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl\" (UID: \"b02fadf0-b23a-4006-a0d8-7669d9565d7b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl" Nov 01 06:13:28.092869 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.092751 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b02fadf0-b23a-4006-a0d8-7669d9565d7b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl\" (UID: \"b02fadf0-b23a-4006-a0d8-7669d9565d7b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl" Nov 01 06:13:28.093337 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.093135 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b02fadf0-b23a-4006-a0d8-7669d9565d7b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl\" (UID: \"b02fadf0-b23a-4006-a0d8-7669d9565d7b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl" Nov 01 06:13:28.093467 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.093401 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d2abea4-3ca9-4866-831d-644af9c391b1-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7\" (UID: \"3d2abea4-3ca9-4866-831d-644af9c391b1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7" Nov 01 06:13:28.093658 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.093571 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d2abea4-3ca9-4866-831d-644af9c391b1-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7\" (UID: \"3d2abea4-3ca9-4866-831d-644af9c391b1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7" Nov 01 06:13:28.128253 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.127978 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv" Nov 01 06:13:28.153860 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.152469 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9fxv\" (UniqueName: \"kubernetes.io/projected/b02fadf0-b23a-4006-a0d8-7669d9565d7b-kube-api-access-s9fxv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl\" (UID: \"b02fadf0-b23a-4006-a0d8-7669d9565d7b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl" Nov 01 06:13:28.154463 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.154201 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc" Nov 01 06:13:28.159147 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.158917 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdntw\" (UniqueName: \"kubernetes.io/projected/3d2abea4-3ca9-4866-831d-644af9c391b1-kube-api-access-jdntw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7\" (UID: \"3d2abea4-3ca9-4866-831d-644af9c391b1\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7" Nov 01 06:13:28.221163 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.220785 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7" Nov 01 06:13:28.227192 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.225777 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl" Nov 01 06:13:28.251417 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.251323 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" event={"ID":"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a","Type":"ContainerStarted","Data":"57e7b59a3a11095784e4703354c530af84fec923fc60b6404fc1e8171e876225"} Nov 01 06:13:28.309649 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.309542 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7"] Nov 01 06:13:28.566669 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.566637 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv"] Nov 01 06:13:28.569812 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:13:28.569772 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f7ef2f_8d4b_4d84_aab9_1bbdb7c370b8.slice/crio-62709f8fa201ddff024d05c2c3f2267061db7f9b6e67312b82f40896c756cf0e WatchSource:0}: Error finding container 62709f8fa201ddff024d05c2c3f2267061db7f9b6e67312b82f40896c756cf0e: Status 404 returned error can't find the container with id 62709f8fa201ddff024d05c2c3f2267061db7f9b6e67312b82f40896c756cf0e Nov 01 06:13:28.590422 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.590353 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc"] Nov 01 06:13:28.651206 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.651170 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7"] Nov 01 06:13:28.656701 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:13:28.656670 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2abea4_3ca9_4866_831d_644af9c391b1.slice/crio-cdc332c0eda8e0536dd9effbed9828480a87febc567150500b96748363cd5c69 WatchSource:0}: Error finding container cdc332c0eda8e0536dd9effbed9828480a87febc567150500b96748363cd5c69: Status 404 returned error can't find the container with id cdc332c0eda8e0536dd9effbed9828480a87febc567150500b96748363cd5c69 Nov 01 06:13:28.668422 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:28.668389 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl"] Nov 01 06:13:28.684140 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:13:28.684068 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02fadf0_b23a_4006_a0d8_7669d9565d7b.slice/crio-f1971977ecf48d6fe7af29e7c472bc7dea2dbc3129a5bdea4f4f851bbab0bca6 WatchSource:0}: Error finding container f1971977ecf48d6fe7af29e7c472bc7dea2dbc3129a5bdea4f4f851bbab0bca6: Status 404 returned error can't find the container with id f1971977ecf48d6fe7af29e7c472bc7dea2dbc3129a5bdea4f4f851bbab0bca6 Nov 01 06:13:29.265650 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:29.265606 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl" event={"ID":"b02fadf0-b23a-4006-a0d8-7669d9565d7b","Type":"ContainerStarted","Data":"f1971977ecf48d6fe7af29e7c472bc7dea2dbc3129a5bdea4f4f851bbab0bca6"} Nov 01 06:13:29.267563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:29.267529 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7" event={"ID":"3d2abea4-3ca9-4866-831d-644af9c391b1","Type":"ContainerStarted","Data":"cdc332c0eda8e0536dd9effbed9828480a87febc567150500b96748363cd5c69"} Nov 01 06:13:29.269582 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:29.269555 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv" event={"ID":"75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8","Type":"ContainerStarted","Data":"62709f8fa201ddff024d05c2c3f2267061db7f9b6e67312b82f40896c756cf0e"} Nov 01 06:13:29.272145 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:29.272117 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" event={"ID":"28f52bd7-a41a-4af8-9f66-911b3fe24b58","Type":"ContainerStarted","Data":"6151d3b8bf83592c92ad26de6212b2fd8815a8c993163831b785d3060ebd8416"} Nov 01 06:13:29.275019 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:29.274994 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc" event={"ID":"e102dd21-a348-4c82-8cc1-3938b7e68387","Type":"ContainerStarted","Data":"c71567dbaaa2813657311f0d5d72a46045501149267c3244a7c507d026868db7"} Nov 01 06:13:31.167725 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:31.167667 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:13:35.246260 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:35.246206 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:13:35.433997 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:35.433895 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:13:36.811493 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:13:36.811450 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:13:41.499638 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:41.499599 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:13:41.500131 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:41.499715 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:13:41.500131 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:41.499766 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:13:41.550393 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:41.550347 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:13:41.550574 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:41.550447 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:13:41.550574 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:41.550491 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:13:42.480146 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:42.480103 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" event={"ID":"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a","Type":"ContainerStarted","Data":"2c03f0b7f2c143b073fff854d013af9f7a5d652c62311a91e8b65a8ebfc7f3fd"} Nov 01 06:13:42.480448 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:42.480210 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:42.481138 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:42.481070 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:42.482697 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:42.482668 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8ckbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 01 06:13:42.482927 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:42.482727 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" podUID="bf4f43cd-9da8-4bd6-9b49-9c7329b5480a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 01 06:13:42.485317 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:42.484676 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" event={"ID":"28f52bd7-a41a-4af8-9f66-911b3fe24b58","Type":"ContainerStarted","Data":"8431fa9827e771b71eefa0f30ce23caa9b3eda91e3b93f2ff579180d2281f11e"} Nov 01 06:13:42.689887 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:42.688193 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" podStartSLOduration=2.170763436 podStartE2EDuration="15.688171747s" totalImagesPullingTime="13.517408311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:13:42.646617617 +0000 UTC m=+232.410417478" watchObservedRunningTime="2026-11-01 06:13:42.688171747 +0000 UTC m=+232.451971605" Nov 01 06:13:42.773727 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:42.773566 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" podStartSLOduration=2.545558217 podStartE2EDuration="15.773543522s" totalImagesPullingTime="13.227985305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:13:42.773246762 +0000 UTC m=+232.537046627" watchObservedRunningTime="2026-11-01 06:13:42.773543522 +0000 UTC m=+232.537343380" Nov 01 06:13:42.786007 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:42.785972 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:42.786943 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:42.786919 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:42.791693 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:42.791660 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:43.489488 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:43.487725 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:43.503776 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:43.501004 4385 generic.go:356] "Generic (PLEG): container finished" podID="05ab5c46-8257-4e49-b936-c83eb203b08f" containerID="470ea9e335c3d1ff377b967681c5db61ba8efecf66aec9284fdf2fc0986625dc" exitCode=0 Nov 01 06:13:43.503776 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:43.502759 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc" event={"ID":"05ab5c46-8257-4e49-b936-c83eb203b08f","Type":"ContainerDied","Data":"470ea9e335c3d1ff377b967681c5db61ba8efecf66aec9284fdf2fc0986625dc"} Nov 01 06:13:43.506341 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:43.506072 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7" Nov 01 06:13:43.507586 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:43.507453 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:13:45.518800 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:45.518767 4385 generic.go:356] "Generic (PLEG): container finished" podID="3d2abea4-3ca9-4866-831d-644af9c391b1" containerID="6e5258015419de133ca9f2797a82a8a45ae35dbe98568ed0fa7bc17f694abf8f" exitCode=0 Nov 01 06:13:45.519267 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:45.518873 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7" event={"ID":"3d2abea4-3ca9-4866-831d-644af9c391b1","Type":"ContainerDied","Data":"6e5258015419de133ca9f2797a82a8a45ae35dbe98568ed0fa7bc17f694abf8f"} Nov 01 06:13:45.522034 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:45.521555 4385 generic.go:356] "Generic (PLEG): container finished" podID="75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8" containerID="4b3db12cd4b83f783f7faf98b99dd6ac3ffda9728fd875dd00fc121310b67a5e" exitCode=0 Nov 01 06:13:45.522034 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:45.521890 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv" event={"ID":"75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8","Type":"ContainerDied","Data":"4b3db12cd4b83f783f7faf98b99dd6ac3ffda9728fd875dd00fc121310b67a5e"} Nov 01 06:13:45.524729 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:45.524317 4385 generic.go:356] "Generic (PLEG): container finished" podID="77c51a13-98cf-45ab-a049-bac31d7cada0" containerID="e55e90bd8c325a31c87d0a555835facdc1b03e1c16a517c1c27e0089297d8495" exitCode=0 Nov 01 06:13:45.524729 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:45.524375 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh" event={"ID":"77c51a13-98cf-45ab-a049-bac31d7cada0","Type":"ContainerDied","Data":"e55e90bd8c325a31c87d0a555835facdc1b03e1c16a517c1c27e0089297d8495"} Nov 01 06:13:46.532241 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:46.532165 4385 generic.go:356] "Generic (PLEG): container finished" podID="e102dd21-a348-4c82-8cc1-3938b7e68387" containerID="c03a65623012b50f191d3947b3c5b0785919452010b5ff3a5f7ca76d03df1ee6" exitCode=0 Nov 01 06:13:46.532694 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:46.532292 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc" event={"ID":"e102dd21-a348-4c82-8cc1-3938b7e68387","Type":"ContainerDied","Data":"c03a65623012b50f191d3947b3c5b0785919452010b5ff3a5f7ca76d03df1ee6"} Nov 01 06:13:46.534153 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:46.534128 4385 generic.go:356] "Generic (PLEG): container finished" podID="c591503c-5b74-45c7-a3f5-13cbecb15650" containerID="6bcb9ad77252abce910b8cd9f174771e9c8ece5f27c69426a6ba36e8b9db8403" exitCode=0 Nov 01 06:13:46.534234 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:46.534193 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv" event={"ID":"c591503c-5b74-45c7-a3f5-13cbecb15650","Type":"ContainerDied","Data":"6bcb9ad77252abce910b8cd9f174771e9c8ece5f27c69426a6ba36e8b9db8403"} Nov 01 06:13:46.536027 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:46.535928 4385 generic.go:356] "Generic (PLEG): container finished" podID="b02fadf0-b23a-4006-a0d8-7669d9565d7b" containerID="d23d1e9412089abcc9bb3ae162bc9d9d7289f067edc76e5d7a68ae69480aba4c" exitCode=0 Nov 01 06:13:46.536115 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:46.536017 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl" event={"ID":"b02fadf0-b23a-4006-a0d8-7669d9565d7b","Type":"ContainerDied","Data":"d23d1e9412089abcc9bb3ae162bc9d9d7289f067edc76e5d7a68ae69480aba4c"} Nov 01 06:13:46.537851 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:46.537731 4385 generic.go:356] "Generic (PLEG): container finished" podID="2ea4d43f-2d68-435a-adec-081098268d28" containerID="c7edce8e4b180db9de463a8c626ae1b59c2f149b98a7b03ff2ef6beb8d03b51a" exitCode=0 Nov 01 06:13:46.538011 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:46.537988 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl" event={"ID":"2ea4d43f-2d68-435a-adec-081098268d28","Type":"ContainerDied","Data":"c7edce8e4b180db9de463a8c626ae1b59c2f149b98a7b03ff2ef6beb8d03b51a"} Nov 01 06:13:46.539685 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:46.539655 4385 generic.go:356] "Generic (PLEG): container finished" podID="9b1e735a-5598-41d3-b557-5161b194e7d3" containerID="990eef5910419f3f2ac7fd16a1116b2c5b82c66a5c7030a6f0c627eb0672d662" exitCode=0 Nov 01 06:13:46.539758 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:46.539744 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm" event={"ID":"9b1e735a-5598-41d3-b557-5161b194e7d3","Type":"ContainerDied","Data":"990eef5910419f3f2ac7fd16a1116b2c5b82c66a5c7030a6f0c627eb0672d662"} Nov 01 06:13:47.546607 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:47.546571 4385 generic.go:356] "Generic (PLEG): container finished" podID="885a807a-2d27-4f2b-8a04-7a6c1388820b" containerID="9d8da80a3c46236228c1b298e83c532e1f1bc27c3ef3fad3868afeb05f99b501" exitCode=0 Nov 01 06:13:47.547096 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:47.546653 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j" event={"ID":"885a807a-2d27-4f2b-8a04-7a6c1388820b","Type":"ContainerDied","Data":"9d8da80a3c46236228c1b298e83c532e1f1bc27c3ef3fad3868afeb05f99b501"} Nov 01 06:13:47.549226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:47.549198 4385 generic.go:356] "Generic (PLEG): container finished" podID="9306df59-d376-411e-8fa4-6606ed054782" containerID="de8e8312e01cf0a77af1ce4e0aa7d9d4b37912ad921d8f70a66e0bc98f01c950" exitCode=0 Nov 01 06:13:47.549326 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:47.549235 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x" event={"ID":"9306df59-d376-411e-8fa4-6606ed054782","Type":"ContainerDied","Data":"de8e8312e01cf0a77af1ce4e0aa7d9d4b37912ad921d8f70a66e0bc98f01c950"} Nov 01 06:13:48.555489 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.555453 4385 generic.go:356] "Generic (PLEG): container finished" podID="05ab5c46-8257-4e49-b936-c83eb203b08f" containerID="e11f9459a6c5158740cf37a64a78d9fa40a71815b2ac4b87bd73d13dad0f679d" exitCode=0 Nov 01 06:13:48.555979 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.555536 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc" event={"ID":"05ab5c46-8257-4e49-b936-c83eb203b08f","Type":"ContainerDied","Data":"e11f9459a6c5158740cf37a64a78d9fa40a71815b2ac4b87bd73d13dad0f679d"} Nov 01 06:13:48.557449 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.557425 4385 generic.go:356] "Generic (PLEG): container finished" podID="9b1e735a-5598-41d3-b557-5161b194e7d3" containerID="6f92495094939cee47ee49732493bb13c1b27c0bcac75ee79025a8573ad99008" exitCode=0 Nov 01 06:13:48.557559 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.557458 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm" event={"ID":"9b1e735a-5598-41d3-b557-5161b194e7d3","Type":"ContainerDied","Data":"6f92495094939cee47ee49732493bb13c1b27c0bcac75ee79025a8573ad99008"} Nov 01 06:13:48.559601 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.559577 4385 generic.go:356] "Generic (PLEG): container finished" podID="77c51a13-98cf-45ab-a049-bac31d7cada0" containerID="5328d6800f58177186c8bd13eeebed906cf4491102f49dad52aa97e806b21355" exitCode=0 Nov 01 06:13:48.559703 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.559666 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh" event={"ID":"77c51a13-98cf-45ab-a049-bac31d7cada0","Type":"ContainerDied","Data":"5328d6800f58177186c8bd13eeebed906cf4491102f49dad52aa97e806b21355"} Nov 01 06:13:48.561500 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.561483 4385 generic.go:356] "Generic (PLEG): container finished" podID="885a807a-2d27-4f2b-8a04-7a6c1388820b" containerID="859b67e64d67bda5b81a1153149ed3c0b1a88cf18037af9afad5c76a915194f9" exitCode=0 Nov 01 06:13:48.561592 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.561582 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j" event={"ID":"885a807a-2d27-4f2b-8a04-7a6c1388820b","Type":"ContainerDied","Data":"859b67e64d67bda5b81a1153149ed3c0b1a88cf18037af9afad5c76a915194f9"} Nov 01 06:13:48.563401 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.563384 4385 generic.go:356] "Generic (PLEG): container finished" podID="e102dd21-a348-4c82-8cc1-3938b7e68387" containerID="04b53a80d5de96e659df05072de7d520e35d2ffdf12f677811688f05c8f1500f" exitCode=0 Nov 01 06:13:48.563482 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.563435 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc" event={"ID":"e102dd21-a348-4c82-8cc1-3938b7e68387","Type":"ContainerDied","Data":"04b53a80d5de96e659df05072de7d520e35d2ffdf12f677811688f05c8f1500f"} Nov 01 06:13:48.565289 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.565266 4385 generic.go:356] "Generic (PLEG): container finished" podID="c591503c-5b74-45c7-a3f5-13cbecb15650" containerID="2c486a02daeef95651ded4e9da6620b7e07077ecde64037cb5a573996e3d3695" exitCode=0 Nov 01 06:13:48.565395 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.565345 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv" event={"ID":"c591503c-5b74-45c7-a3f5-13cbecb15650","Type":"ContainerDied","Data":"2c486a02daeef95651ded4e9da6620b7e07077ecde64037cb5a573996e3d3695"} Nov 01 06:13:48.567100 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.567077 4385 generic.go:356] "Generic (PLEG): container finished" podID="e49365a6-eff4-4d06-a968-9a05e25a70c0" containerID="559bdffce8a394d7efbb449b58cec0fb0459ac43b58051ffb2f5fc815fa44795" exitCode=0 Nov 01 06:13:48.567226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.567144 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9" event={"ID":"e49365a6-eff4-4d06-a968-9a05e25a70c0","Type":"ContainerDied","Data":"559bdffce8a394d7efbb449b58cec0fb0459ac43b58051ffb2f5fc815fa44795"} Nov 01 06:13:48.569184 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.569163 4385 generic.go:356] "Generic (PLEG): container finished" podID="b02fadf0-b23a-4006-a0d8-7669d9565d7b" containerID="96893b3bda68b77f0ef8558583eebaa4eacbff84fa5fbc3ace477330bc538850" exitCode=0 Nov 01 06:13:48.569279 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.569231 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl" event={"ID":"b02fadf0-b23a-4006-a0d8-7669d9565d7b","Type":"ContainerDied","Data":"96893b3bda68b77f0ef8558583eebaa4eacbff84fa5fbc3ace477330bc538850"} Nov 01 06:13:48.571585 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.571478 4385 generic.go:356] "Generic (PLEG): container finished" podID="2ea4d43f-2d68-435a-adec-081098268d28" containerID="bfd75706f395076f54fe91ddacd83e65682a8463083d3522db6254e5eb1d5843" exitCode=0 Nov 01 06:13:48.571585 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.571562 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl" event={"ID":"2ea4d43f-2d68-435a-adec-081098268d28","Type":"ContainerDied","Data":"bfd75706f395076f54fe91ddacd83e65682a8463083d3522db6254e5eb1d5843"} Nov 01 06:13:48.573822 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.573804 4385 generic.go:356] "Generic (PLEG): container finished" podID="3d2abea4-3ca9-4866-831d-644af9c391b1" containerID="9f432bd5e0666e52cb7145fa23e266b9298a4825f588905c20a0608b0375cdab" exitCode=0 Nov 01 06:13:48.573922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.573904 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7" event={"ID":"3d2abea4-3ca9-4866-831d-644af9c391b1","Type":"ContainerDied","Data":"9f432bd5e0666e52cb7145fa23e266b9298a4825f588905c20a0608b0375cdab"} Nov 01 06:13:48.576282 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.576263 4385 generic.go:356] "Generic (PLEG): container finished" podID="75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8" containerID="e7d0ee0d9b56108fb1741f94269f86033ae71712ae26e4631354df6de2171374" exitCode=0 Nov 01 06:13:48.576361 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:48.576305 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv" event={"ID":"75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8","Type":"ContainerDied","Data":"e7d0ee0d9b56108fb1741f94269f86033ae71712ae26e4631354df6de2171374"} Nov 01 06:13:49.594165 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:49.593271 4385 generic.go:356] "Generic (PLEG): container finished" podID="e49365a6-eff4-4d06-a968-9a05e25a70c0" containerID="01e3392efb7517e398cf960013a1d4704ac368f2fa507102039727cd31a7448f" exitCode=0 Nov 01 06:13:49.594165 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:49.593422 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9" event={"ID":"e49365a6-eff4-4d06-a968-9a05e25a70c0","Type":"ContainerDied","Data":"01e3392efb7517e398cf960013a1d4704ac368f2fa507102039727cd31a7448f"} Nov 01 06:13:57.723678 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:57.723627 4385 generic.go:356] "Generic (PLEG): container finished" podID="9306df59-d376-411e-8fa4-6606ed054782" containerID="c85236a9412069fbf354e8414ea015de7ea0d966c51ec2038ced1ff6be7359cb" exitCode=0 Nov 01 06:13:57.724229 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:57.723700 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x" event={"ID":"9306df59-d376-411e-8fa4-6606ed054782","Type":"ContainerDied","Data":"c85236a9412069fbf354e8414ea015de7ea0d966c51ec2038ced1ff6be7359cb"} Nov 01 06:13:57.956586 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:57.956509 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:13:58.164258 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:13:58.164223 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:06.797209 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:14:06.797091 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:14:14.947692 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:14.947654 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl" event={"ID":"b02fadf0-b23a-4006-a0d8-7669d9565d7b","Type":"ContainerStarted","Data":"5557bef379e8ebc19c2e53275ea9b95a7e7f26fd69fccd593eec6b68e07aaf4b"} Nov 01 06:14:14.953368 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:14.953336 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x" event={"ID":"9306df59-d376-411e-8fa4-6606ed054782","Type":"ContainerStarted","Data":"565d407184b0514102d73df8eb355008c2c8ed676e829a171738130a226671c0"} Nov 01 06:14:15.078544 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:15.078491 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl" podStartSLOduration=4.027556635 podStartE2EDuration="48.078473903s" totalImagesPullingTime="44.050917268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:13:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:14:15.07675113 +0000 UTC m=+264.840550984" watchObservedRunningTime="2026-11-01 06:14:15.078473903 +0000 UTC m=+264.842273760" Nov 01 06:14:15.194509 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:15.193147 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x" podStartSLOduration=4.051389411 podStartE2EDuration="55.19312692s" totalImagesPullingTime="51.141737509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:13:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:14:15.1920713 +0000 UTC m=+264.955871160" watchObservedRunningTime="2026-11-01 06:14:15.19312692 +0000 UTC m=+264.956926781" Nov 01 06:14:15.963566 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:15.963492 4385 generic.go:356] "Generic (PLEG): container finished" podID="b02fadf0-b23a-4006-a0d8-7669d9565d7b" containerID="5557bef379e8ebc19c2e53275ea9b95a7e7f26fd69fccd593eec6b68e07aaf4b" exitCode=0 Nov 01 06:14:15.964035 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:15.963571 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl" event={"ID":"b02fadf0-b23a-4006-a0d8-7669d9565d7b","Type":"ContainerDied","Data":"5557bef379e8ebc19c2e53275ea9b95a7e7f26fd69fccd593eec6b68e07aaf4b"} Nov 01 06:14:15.965625 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:15.965603 4385 generic.go:356] "Generic (PLEG): container finished" podID="9306df59-d376-411e-8fa4-6606ed054782" containerID="565d407184b0514102d73df8eb355008c2c8ed676e829a171738130a226671c0" exitCode=0 Nov 01 06:14:15.965757 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:15.965692 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x" event={"ID":"9306df59-d376-411e-8fa4-6606ed054782","Type":"ContainerDied","Data":"565d407184b0514102d73df8eb355008c2c8ed676e829a171738130a226671c0"} Nov 01 06:14:15.967502 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:15.967478 4385 generic.go:356] "Generic (PLEG): container finished" podID="2ea4d43f-2d68-435a-adec-081098268d28" containerID="9b9980ad4e546f3f99c8b9d9adc6ba102d1c27898a704ce7df1db558e9398126" exitCode=0 Nov 01 06:14:15.967585 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:15.967563 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl" event={"ID":"2ea4d43f-2d68-435a-adec-081098268d28","Type":"ContainerDied","Data":"9b9980ad4e546f3f99c8b9d9adc6ba102d1c27898a704ce7df1db558e9398126"} Nov 01 06:14:15.969461 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:15.969438 4385 generic.go:356] "Generic (PLEG): container finished" podID="75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8" containerID="d8d79156b9105a97fb61b9e588ed32f4a739dfcf28bcbebd64a2f252deff4f76" exitCode=0 Nov 01 06:14:15.969602 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:15.969581 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv" event={"ID":"75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8","Type":"ContainerDied","Data":"d8d79156b9105a97fb61b9e588ed32f4a739dfcf28bcbebd64a2f252deff4f76"} Nov 01 06:14:15.971750 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:15.971729 4385 generic.go:356] "Generic (PLEG): container finished" podID="77c51a13-98cf-45ab-a049-bac31d7cada0" containerID="3d99f460c530a5fdf5a547cec58a9b844e78868930a272e072a39ce3ce8df11a" exitCode=0 Nov 01 06:14:15.971866 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:15.971814 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh" event={"ID":"77c51a13-98cf-45ab-a049-bac31d7cada0","Type":"ContainerDied","Data":"3d99f460c530a5fdf5a547cec58a9b844e78868930a272e072a39ce3ce8df11a"} Nov 01 06:14:15.973933 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:15.973914 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc" event={"ID":"e102dd21-a348-4c82-8cc1-3938b7e68387","Type":"ContainerStarted","Data":"df9d2678b3d1339dcd2c4e0e4f352436f09849789ea32e67e6bba0d0c74a6d38"} Nov 01 06:14:16.533098 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:16.533040 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc" podStartSLOduration=3.918070683 podStartE2EDuration="49.533023628s" totalImagesPullingTime="45.614952945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:13:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:14:16.532347934 +0000 UTC m=+266.296147792" watchObservedRunningTime="2026-11-01 06:14:16.533023628 +0000 UTC m=+266.296823538" Nov 01 06:14:16.981176 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:16.981137 4385 generic.go:356] "Generic (PLEG): container finished" podID="05ab5c46-8257-4e49-b936-c83eb203b08f" containerID="4b6f6f88f75d34723ccbfe81822e64ec4ef9db352e55f0c265a65aee47a0d702" exitCode=0 Nov 01 06:14:16.981557 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:16.981210 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc" event={"ID":"05ab5c46-8257-4e49-b936-c83eb203b08f","Type":"ContainerDied","Data":"4b6f6f88f75d34723ccbfe81822e64ec4ef9db352e55f0c265a65aee47a0d702"} Nov 01 06:14:16.983295 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:16.983273 4385 generic.go:356] "Generic (PLEG): container finished" podID="9b1e735a-5598-41d3-b557-5161b194e7d3" containerID="737f6851d6b3e1c6cf90d1cd853d02bc85214d757050fae283e16b4f6da652e2" exitCode=0 Nov 01 06:14:16.983424 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:16.983357 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm" event={"ID":"9b1e735a-5598-41d3-b557-5161b194e7d3","Type":"ContainerDied","Data":"737f6851d6b3e1c6cf90d1cd853d02bc85214d757050fae283e16b4f6da652e2"} Nov 01 06:14:16.985308 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:16.985288 4385 generic.go:356] "Generic (PLEG): container finished" podID="e102dd21-a348-4c82-8cc1-3938b7e68387" containerID="df9d2678b3d1339dcd2c4e0e4f352436f09849789ea32e67e6bba0d0c74a6d38" exitCode=0 Nov 01 06:14:16.985394 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:16.985335 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc" event={"ID":"e102dd21-a348-4c82-8cc1-3938b7e68387","Type":"ContainerDied","Data":"df9d2678b3d1339dcd2c4e0e4f352436f09849789ea32e67e6bba0d0c74a6d38"} Nov 01 06:14:16.987245 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:16.987227 4385 generic.go:356] "Generic (PLEG): container finished" podID="c591503c-5b74-45c7-a3f5-13cbecb15650" containerID="7e3394ffb2ccfc209e02d05003feff0f2247f43e0f3e3a9b2789488e5f5394fa" exitCode=0 Nov 01 06:14:16.987335 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:16.987309 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv" event={"ID":"c591503c-5b74-45c7-a3f5-13cbecb15650","Type":"ContainerDied","Data":"7e3394ffb2ccfc209e02d05003feff0f2247f43e0f3e3a9b2789488e5f5394fa"} Nov 01 06:14:16.989269 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:16.989251 4385 generic.go:356] "Generic (PLEG): container finished" podID="e49365a6-eff4-4d06-a968-9a05e25a70c0" containerID="39abfe3a89208dfc08fd22d1bbcd4156cff6781b96e252d4c3f8882e24e5a052" exitCode=0 Nov 01 06:14:16.989370 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:16.989277 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9" event={"ID":"e49365a6-eff4-4d06-a968-9a05e25a70c0","Type":"ContainerDied","Data":"39abfe3a89208dfc08fd22d1bbcd4156cff6781b96e252d4c3f8882e24e5a052"} Nov 01 06:14:16.991243 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:16.991160 4385 generic.go:356] "Generic (PLEG): container finished" podID="3d2abea4-3ca9-4866-831d-644af9c391b1" containerID="05985b030ef531e21507e6a1463da24a15ad7feaa00a1c4508b45f0dd91d90ee" exitCode=0 Nov 01 06:14:16.991308 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:16.991253 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7" event={"ID":"3d2abea4-3ca9-4866-831d-644af9c391b1","Type":"ContainerDied","Data":"05985b030ef531e21507e6a1463da24a15ad7feaa00a1c4508b45f0dd91d90ee"} Nov 01 06:14:17.135091 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.135063 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x" Nov 01 06:14:17.218960 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.218929 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh" Nov 01 06:14:17.227550 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.227528 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv" Nov 01 06:14:17.230889 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.230871 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl" Nov 01 06:14:17.234061 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.234041 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl" Nov 01 06:14:17.241581 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.241562 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9306df59-d376-411e-8fa4-6606ed054782-util\" (UniqueName: \"kubernetes.io/empty-dir/9306df59-d376-411e-8fa4-6606ed054782-util\") pod \"9306df59-d376-411e-8fa4-6606ed054782\" (UID: \"9306df59-d376-411e-8fa4-6606ed054782\") " Nov 01 06:14:17.241692 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.241677 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9306df59-d376-411e-8fa4-6606ed054782-kube-api-access-cclxx\" (UniqueName: \"kubernetes.io/projected/9306df59-d376-411e-8fa4-6606ed054782-kube-api-access-cclxx\") pod \"9306df59-d376-411e-8fa4-6606ed054782\" (UID: \"9306df59-d376-411e-8fa4-6606ed054782\") " Nov 01 06:14:17.241737 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.241705 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9306df59-d376-411e-8fa4-6606ed054782-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9306df59-d376-411e-8fa4-6606ed054782-bundle\") pod \"9306df59-d376-411e-8fa4-6606ed054782\" (UID: \"9306df59-d376-411e-8fa4-6606ed054782\") " Nov 01 06:14:17.242517 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.242433 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9306df59-d376-411e-8fa4-6606ed054782-bundle" pod "9306df59-d376-411e-8fa4-6606ed054782" (UID: "9306df59-d376-411e-8fa4-6606ed054782"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:17.244056 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.244034 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9306df59-d376-411e-8fa4-6606ed054782-kube-api-access-cclxx" pod "9306df59-d376-411e-8fa4-6606ed054782" (UID: "9306df59-d376-411e-8fa4-6606ed054782"). InnerVolumeSpecName "kube-api-access-cclxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:17.247428 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.247407 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9306df59-d376-411e-8fa4-6606ed054782-util" pod "9306df59-d376-411e-8fa4-6606ed054782" (UID: "9306df59-d376-411e-8fa4-6606ed054782"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:17.342780 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.342749 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea4d43f-2d68-435a-adec-081098268d28-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ea4d43f-2d68-435a-adec-081098268d28-bundle\") pod \"2ea4d43f-2d68-435a-adec-081098268d28\" (UID: \"2ea4d43f-2d68-435a-adec-081098268d28\") " Nov 01 06:14:17.342961 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.342796 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8-kube-api-access-nppqk\" (UniqueName: \"kubernetes.io/projected/75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8-kube-api-access-nppqk\") pod \"75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8\" (UID: \"75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8\") " Nov 01 06:14:17.342961 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.342824 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c51a13-98cf-45ab-a049-bac31d7cada0-util\" (UniqueName: \"kubernetes.io/empty-dir/77c51a13-98cf-45ab-a049-bac31d7cada0-util\") pod \"77c51a13-98cf-45ab-a049-bac31d7cada0\" (UID: \"77c51a13-98cf-45ab-a049-bac31d7cada0\") " Nov 01 06:14:17.342961 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.342907 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea4d43f-2d68-435a-adec-081098268d28-kube-api-access-jvt6c\" (UniqueName: \"kubernetes.io/projected/2ea4d43f-2d68-435a-adec-081098268d28-kube-api-access-jvt6c\") pod \"2ea4d43f-2d68-435a-adec-081098268d28\" (UID: \"2ea4d43f-2d68-435a-adec-081098268d28\") " Nov 01 06:14:17.342961 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.342949 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8-bundle\") pod \"75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8\" (UID: \"75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8\") " Nov 01 06:14:17.343165 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.342979 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c51a13-98cf-45ab-a049-bac31d7cada0-kube-api-access-lnc8q\" (UniqueName: \"kubernetes.io/projected/77c51a13-98cf-45ab-a049-bac31d7cada0-kube-api-access-lnc8q\") pod \"77c51a13-98cf-45ab-a049-bac31d7cada0\" (UID: \"77c51a13-98cf-45ab-a049-bac31d7cada0\") " Nov 01 06:14:17.343165 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.343042 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8-util\" (UniqueName: \"kubernetes.io/empty-dir/75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8-util\") pod \"75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8\" (UID: \"75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8\") " Nov 01 06:14:17.343165 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.343083 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea4d43f-2d68-435a-adec-081098268d28-util\" (UniqueName: \"kubernetes.io/empty-dir/2ea4d43f-2d68-435a-adec-081098268d28-util\") pod \"2ea4d43f-2d68-435a-adec-081098268d28\" (UID: \"2ea4d43f-2d68-435a-adec-081098268d28\") " Nov 01 06:14:17.343165 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.343127 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b02fadf0-b23a-4006-a0d8-7669d9565d7b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b02fadf0-b23a-4006-a0d8-7669d9565d7b-bundle\") pod \"b02fadf0-b23a-4006-a0d8-7669d9565d7b\" (UID: \"b02fadf0-b23a-4006-a0d8-7669d9565d7b\") " Nov 01 06:14:17.343165 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.343156 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77c51a13-98cf-45ab-a049-bac31d7cada0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/77c51a13-98cf-45ab-a049-bac31d7cada0-bundle\") pod \"77c51a13-98cf-45ab-a049-bac31d7cada0\" (UID: \"77c51a13-98cf-45ab-a049-bac31d7cada0\") " Nov 01 06:14:17.343414 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.343184 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b02fadf0-b23a-4006-a0d8-7669d9565d7b-util\" (UniqueName: \"kubernetes.io/empty-dir/b02fadf0-b23a-4006-a0d8-7669d9565d7b-util\") pod \"b02fadf0-b23a-4006-a0d8-7669d9565d7b\" (UID: \"b02fadf0-b23a-4006-a0d8-7669d9565d7b\") " Nov 01 06:14:17.343414 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.343220 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02fadf0-b23a-4006-a0d8-7669d9565d7b-kube-api-access-s9fxv\" (UniqueName: \"kubernetes.io/projected/b02fadf0-b23a-4006-a0d8-7669d9565d7b-kube-api-access-s9fxv\") pod \"b02fadf0-b23a-4006-a0d8-7669d9565d7b\" (UID: \"b02fadf0-b23a-4006-a0d8-7669d9565d7b\") " Nov 01 06:14:17.343526 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.343502 4385 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9306df59-d376-411e-8fa4-6606ed054782-util\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:17.343581 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.343529 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cclxx\" (UniqueName: \"kubernetes.io/projected/9306df59-d376-411e-8fa4-6606ed054782-kube-api-access-cclxx\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:17.343581 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.343545 4385 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9306df59-d376-411e-8fa4-6606ed054782-bundle\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:17.343581 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.343550 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea4d43f-2d68-435a-adec-081098268d28-bundle" pod "2ea4d43f-2d68-435a-adec-081098268d28" (UID: "2ea4d43f-2d68-435a-adec-081098268d28"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:17.345567 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.343756 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8-bundle" pod "75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8" (UID: "75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:17.345567 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.344872 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02fadf0-b23a-4006-a0d8-7669d9565d7b-bundle" pod "b02fadf0-b23a-4006-a0d8-7669d9565d7b" (UID: "b02fadf0-b23a-4006-a0d8-7669d9565d7b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:17.345567 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.344935 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c51a13-98cf-45ab-a049-bac31d7cada0-bundle" pod "77c51a13-98cf-45ab-a049-bac31d7cada0" (UID: "77c51a13-98cf-45ab-a049-bac31d7cada0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:17.345791 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.345661 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea4d43f-2d68-435a-adec-081098268d28-kube-api-access-jvt6c" pod "2ea4d43f-2d68-435a-adec-081098268d28" (UID: "2ea4d43f-2d68-435a-adec-081098268d28"). InnerVolumeSpecName "kube-api-access-jvt6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:17.345890 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.345856 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c51a13-98cf-45ab-a049-bac31d7cada0-kube-api-access-lnc8q" pod "77c51a13-98cf-45ab-a049-bac31d7cada0" (UID: "77c51a13-98cf-45ab-a049-bac31d7cada0"). InnerVolumeSpecName "kube-api-access-lnc8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:17.346011 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.345990 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02fadf0-b23a-4006-a0d8-7669d9565d7b-kube-api-access-s9fxv" pod "b02fadf0-b23a-4006-a0d8-7669d9565d7b" (UID: "b02fadf0-b23a-4006-a0d8-7669d9565d7b"). InnerVolumeSpecName "kube-api-access-s9fxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:17.346064 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.346051 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8-kube-api-access-nppqk" pod "75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8" (UID: "75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8"). InnerVolumeSpecName "kube-api-access-nppqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:17.350211 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.350185 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c51a13-98cf-45ab-a049-bac31d7cada0-util" pod "77c51a13-98cf-45ab-a049-bac31d7cada0" (UID: "77c51a13-98cf-45ab-a049-bac31d7cada0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:17.350407 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.350384 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8-util" pod "75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8" (UID: "75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:17.351913 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.351895 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea4d43f-2d68-435a-adec-081098268d28-util" pod "2ea4d43f-2d68-435a-adec-081098268d28" (UID: "2ea4d43f-2d68-435a-adec-081098268d28"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:17.351970 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.351926 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02fadf0-b23a-4006-a0d8-7669d9565d7b-util" pod "b02fadf0-b23a-4006-a0d8-7669d9565d7b" (UID: "b02fadf0-b23a-4006-a0d8-7669d9565d7b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:17.444614 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.444576 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvt6c\" (UniqueName: \"kubernetes.io/projected/2ea4d43f-2d68-435a-adec-081098268d28-kube-api-access-jvt6c\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:17.444614 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.444609 4385 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8-bundle\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:17.444614 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.444620 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnc8q\" (UniqueName: \"kubernetes.io/projected/77c51a13-98cf-45ab-a049-bac31d7cada0-kube-api-access-lnc8q\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:17.444974 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.444710 4385 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8-util\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:17.444974 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.444720 4385 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ea4d43f-2d68-435a-adec-081098268d28-util\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:17.444974 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.444728 4385 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b02fadf0-b23a-4006-a0d8-7669d9565d7b-bundle\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:17.444974 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.444736 4385 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/77c51a13-98cf-45ab-a049-bac31d7cada0-bundle\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:17.444974 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.444749 4385 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b02fadf0-b23a-4006-a0d8-7669d9565d7b-util\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:17.444974 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.444760 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9fxv\" (UniqueName: \"kubernetes.io/projected/b02fadf0-b23a-4006-a0d8-7669d9565d7b-kube-api-access-s9fxv\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:17.444974 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.444768 4385 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ea4d43f-2d68-435a-adec-081098268d28-bundle\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:17.444974 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.444778 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nppqk\" (UniqueName: \"kubernetes.io/projected/75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8-kube-api-access-nppqk\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:17.444974 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.444785 4385 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/77c51a13-98cf-45ab-a049-bac31d7cada0-util\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:17.997461 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.997424 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh" event={"ID":"77c51a13-98cf-45ab-a049-bac31d7cada0","Type":"ContainerDied","Data":"c93d144105f00585ce4a719724caf0eae7fd1505899e6f67b24320e957d65eb0"} Nov 01 06:14:17.997988 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.997486 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c93d144105f00585ce4a719724caf0eae7fd1505899e6f67b24320e957d65eb0" Nov 01 06:14:17.997988 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:17.997433 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wdcfh" Nov 01 06:14:18.000535 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.000506 4385 generic.go:356] "Generic (PLEG): container finished" podID="885a807a-2d27-4f2b-8a04-7a6c1388820b" containerID="f07327b68f714870fa60cf862184b4320b28d02dfd7786f81166c29d2b3ff75f" exitCode=0 Nov 01 06:14:18.000796 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.000567 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j" event={"ID":"885a807a-2d27-4f2b-8a04-7a6c1388820b","Type":"ContainerDied","Data":"f07327b68f714870fa60cf862184b4320b28d02dfd7786f81166c29d2b3ff75f"} Nov 01 06:14:18.003413 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.003388 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl" event={"ID":"b02fadf0-b23a-4006-a0d8-7669d9565d7b","Type":"ContainerDied","Data":"f1971977ecf48d6fe7af29e7c472bc7dea2dbc3129a5bdea4f4f851bbab0bca6"} Nov 01 06:14:18.003593 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.003567 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1971977ecf48d6fe7af29e7c472bc7dea2dbc3129a5bdea4f4f851bbab0bca6" Nov 01 06:14:18.003593 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.003400 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4dqvl" Nov 01 06:14:18.005535 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.005517 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x" Nov 01 06:14:18.005624 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.005519 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c82g7x" event={"ID":"9306df59-d376-411e-8fa4-6606ed054782","Type":"ContainerDied","Data":"295abd59bdadadf7018dd006d8799b2fec39c31bde6f7188549773a5bb0ce55a"} Nov 01 06:14:18.005691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.005647 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="295abd59bdadadf7018dd006d8799b2fec39c31bde6f7188549773a5bb0ce55a" Nov 01 06:14:18.009412 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.009379 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl" event={"ID":"2ea4d43f-2d68-435a-adec-081098268d28","Type":"ContainerDied","Data":"9259989ada70648360eb7ff0a948df3a87958647bb3cc111d6dac6c3eb585938"} Nov 01 06:14:18.009502 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.009436 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9259989ada70648360eb7ff0a948df3a87958647bb3cc111d6dac6c3eb585938" Nov 01 06:14:18.009502 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.009396 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jm9dl" Nov 01 06:14:18.011635 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.011573 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv" Nov 01 06:14:18.011635 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.011601 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acghrv" event={"ID":"75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8","Type":"ContainerDied","Data":"62709f8fa201ddff024d05c2c3f2267061db7f9b6e67312b82f40896c756cf0e"} Nov 01 06:14:18.011778 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.011648 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62709f8fa201ddff024d05c2c3f2267061db7f9b6e67312b82f40896c756cf0e" Nov 01 06:14:18.179214 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.179188 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv" Nov 01 06:14:18.218750 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.218719 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc" Nov 01 06:14:18.250105 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.250041 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9" Nov 01 06:14:18.258006 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.257967 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c591503c-5b74-45c7-a3f5-13cbecb15650-util\" (UniqueName: \"kubernetes.io/empty-dir/c591503c-5b74-45c7-a3f5-13cbecb15650-util\") pod \"c591503c-5b74-45c7-a3f5-13cbecb15650\" (UID: \"c591503c-5b74-45c7-a3f5-13cbecb15650\") " Nov 01 06:14:18.258166 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.258020 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c591503c-5b74-45c7-a3f5-13cbecb15650-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c591503c-5b74-45c7-a3f5-13cbecb15650-bundle\") pod \"c591503c-5b74-45c7-a3f5-13cbecb15650\" (UID: \"c591503c-5b74-45c7-a3f5-13cbecb15650\") " Nov 01 06:14:18.258166 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.258073 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c591503c-5b74-45c7-a3f5-13cbecb15650-kube-api-access-mjlfv\" (UniqueName: \"kubernetes.io/projected/c591503c-5b74-45c7-a3f5-13cbecb15650-kube-api-access-mjlfv\") pod \"c591503c-5b74-45c7-a3f5-13cbecb15650\" (UID: \"c591503c-5b74-45c7-a3f5-13cbecb15650\") " Nov 01 06:14:18.259034 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.259002 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c591503c-5b74-45c7-a3f5-13cbecb15650-bundle" pod "c591503c-5b74-45c7-a3f5-13cbecb15650" (UID: "c591503c-5b74-45c7-a3f5-13cbecb15650"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:18.263502 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.263471 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c591503c-5b74-45c7-a3f5-13cbecb15650-kube-api-access-mjlfv" pod "c591503c-5b74-45c7-a3f5-13cbecb15650" (UID: "c591503c-5b74-45c7-a3f5-13cbecb15650"). InnerVolumeSpecName "kube-api-access-mjlfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:18.263803 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.263780 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c591503c-5b74-45c7-a3f5-13cbecb15650-util" pod "c591503c-5b74-45c7-a3f5-13cbecb15650" (UID: "c591503c-5b74-45c7-a3f5-13cbecb15650"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:18.285283 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.285258 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm" Nov 01 06:14:18.287735 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.287717 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc" Nov 01 06:14:18.291346 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.291327 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7" Nov 01 06:14:18.359050 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.359020 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49365a6-eff4-4d06-a968-9a05e25a70c0-kube-api-access-f9x7f\" (UniqueName: \"kubernetes.io/projected/e49365a6-eff4-4d06-a968-9a05e25a70c0-kube-api-access-f9x7f\") pod \"e49365a6-eff4-4d06-a968-9a05e25a70c0\" (UID: \"e49365a6-eff4-4d06-a968-9a05e25a70c0\") " Nov 01 06:14:18.359225 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.359061 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e102dd21-a348-4c82-8cc1-3938b7e68387-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e102dd21-a348-4c82-8cc1-3938b7e68387-bundle\") pod \"e102dd21-a348-4c82-8cc1-3938b7e68387\" (UID: \"e102dd21-a348-4c82-8cc1-3938b7e68387\") " Nov 01 06:14:18.359225 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.359104 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b1e735a-5598-41d3-b557-5161b194e7d3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b1e735a-5598-41d3-b557-5161b194e7d3-bundle\") pod \"9b1e735a-5598-41d3-b557-5161b194e7d3\" (UID: \"9b1e735a-5598-41d3-b557-5161b194e7d3\") " Nov 01 06:14:18.359225 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.359124 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49365a6-eff4-4d06-a968-9a05e25a70c0-util\" (UniqueName: \"kubernetes.io/empty-dir/e49365a6-eff4-4d06-a968-9a05e25a70c0-util\") pod \"e49365a6-eff4-4d06-a968-9a05e25a70c0\" (UID: \"e49365a6-eff4-4d06-a968-9a05e25a70c0\") " Nov 01 06:14:18.359225 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.359141 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2abea4-3ca9-4866-831d-644af9c391b1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d2abea4-3ca9-4866-831d-644af9c391b1-bundle\") pod \"3d2abea4-3ca9-4866-831d-644af9c391b1\" (UID: \"3d2abea4-3ca9-4866-831d-644af9c391b1\") " Nov 01 06:14:18.359225 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.359178 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2abea4-3ca9-4866-831d-644af9c391b1-util\" (UniqueName: \"kubernetes.io/empty-dir/3d2abea4-3ca9-4866-831d-644af9c391b1-util\") pod \"3d2abea4-3ca9-4866-831d-644af9c391b1\" (UID: \"3d2abea4-3ca9-4866-831d-644af9c391b1\") " Nov 01 06:14:18.359225 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.359214 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ab5c46-8257-4e49-b936-c83eb203b08f-kube-api-access-x8gb4\" (UniqueName: \"kubernetes.io/projected/05ab5c46-8257-4e49-b936-c83eb203b08f-kube-api-access-x8gb4\") pod \"05ab5c46-8257-4e49-b936-c83eb203b08f\" (UID: \"05ab5c46-8257-4e49-b936-c83eb203b08f\") " Nov 01 06:14:18.359551 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.359244 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ab5c46-8257-4e49-b936-c83eb203b08f-util\" (UniqueName: \"kubernetes.io/empty-dir/05ab5c46-8257-4e49-b936-c83eb203b08f-util\") pod \"05ab5c46-8257-4e49-b936-c83eb203b08f\" (UID: \"05ab5c46-8257-4e49-b936-c83eb203b08f\") " Nov 01 06:14:18.359551 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.359271 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49365a6-eff4-4d06-a968-9a05e25a70c0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e49365a6-eff4-4d06-a968-9a05e25a70c0-bundle\") pod \"e49365a6-eff4-4d06-a968-9a05e25a70c0\" (UID: \"e49365a6-eff4-4d06-a968-9a05e25a70c0\") " Nov 01 06:14:18.359551 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.359303 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b1e735a-5598-41d3-b557-5161b194e7d3-util\" (UniqueName: \"kubernetes.io/empty-dir/9b1e735a-5598-41d3-b557-5161b194e7d3-util\") pod \"9b1e735a-5598-41d3-b557-5161b194e7d3\" (UID: \"9b1e735a-5598-41d3-b557-5161b194e7d3\") " Nov 01 06:14:18.359551 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.359332 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e102dd21-a348-4c82-8cc1-3938b7e68387-kube-api-access-vdlp7\" (UniqueName: \"kubernetes.io/projected/e102dd21-a348-4c82-8cc1-3938b7e68387-kube-api-access-vdlp7\") pod \"e102dd21-a348-4c82-8cc1-3938b7e68387\" (UID: \"e102dd21-a348-4c82-8cc1-3938b7e68387\") " Nov 01 06:14:18.359551 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.359363 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e102dd21-a348-4c82-8cc1-3938b7e68387-util\" (UniqueName: \"kubernetes.io/empty-dir/e102dd21-a348-4c82-8cc1-3938b7e68387-util\") pod \"e102dd21-a348-4c82-8cc1-3938b7e68387\" (UID: \"e102dd21-a348-4c82-8cc1-3938b7e68387\") " Nov 01 06:14:18.359551 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.359386 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2abea4-3ca9-4866-831d-644af9c391b1-kube-api-access-jdntw\" (UniqueName: \"kubernetes.io/projected/3d2abea4-3ca9-4866-831d-644af9c391b1-kube-api-access-jdntw\") pod \"3d2abea4-3ca9-4866-831d-644af9c391b1\" (UID: \"3d2abea4-3ca9-4866-831d-644af9c391b1\") " Nov 01 06:14:18.359551 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.359432 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b1e735a-5598-41d3-b557-5161b194e7d3-kube-api-access-jpv4g\" (UniqueName: \"kubernetes.io/projected/9b1e735a-5598-41d3-b557-5161b194e7d3-kube-api-access-jpv4g\") pod \"9b1e735a-5598-41d3-b557-5161b194e7d3\" (UID: \"9b1e735a-5598-41d3-b557-5161b194e7d3\") " Nov 01 06:14:18.359551 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.359511 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ab5c46-8257-4e49-b936-c83eb203b08f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/05ab5c46-8257-4e49-b936-c83eb203b08f-bundle\") pod \"05ab5c46-8257-4e49-b936-c83eb203b08f\" (UID: \"05ab5c46-8257-4e49-b936-c83eb203b08f\") " Nov 01 06:14:18.360104 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.359889 4385 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c591503c-5b74-45c7-a3f5-13cbecb15650-util\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:18.360104 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.359912 4385 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c591503c-5b74-45c7-a3f5-13cbecb15650-bundle\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:18.360104 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.359928 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjlfv\" (UniqueName: \"kubernetes.io/projected/c591503c-5b74-45c7-a3f5-13cbecb15650-kube-api-access-mjlfv\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:18.360316 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.360250 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ab5c46-8257-4e49-b936-c83eb203b08f-bundle" pod "05ab5c46-8257-4e49-b936-c83eb203b08f" (UID: "05ab5c46-8257-4e49-b936-c83eb203b08f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:18.361564 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.361541 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49365a6-eff4-4d06-a968-9a05e25a70c0-bundle" pod "e49365a6-eff4-4d06-a968-9a05e25a70c0" (UID: "e49365a6-eff4-4d06-a968-9a05e25a70c0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:18.361883 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.361848 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e102dd21-a348-4c82-8cc1-3938b7e68387-bundle" pod "e102dd21-a348-4c82-8cc1-3938b7e68387" (UID: "e102dd21-a348-4c82-8cc1-3938b7e68387"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:18.362213 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.362194 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2abea4-3ca9-4866-831d-644af9c391b1-bundle" pod "3d2abea4-3ca9-4866-831d-644af9c391b1" (UID: "3d2abea4-3ca9-4866-831d-644af9c391b1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:18.362430 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.362390 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49365a6-eff4-4d06-a968-9a05e25a70c0-kube-api-access-f9x7f" pod "e49365a6-eff4-4d06-a968-9a05e25a70c0" (UID: "e49365a6-eff4-4d06-a968-9a05e25a70c0"). InnerVolumeSpecName "kube-api-access-f9x7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:18.364378 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.363194 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1e735a-5598-41d3-b557-5161b194e7d3-bundle" pod "9b1e735a-5598-41d3-b557-5161b194e7d3" (UID: "9b1e735a-5598-41d3-b557-5161b194e7d3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:18.364565 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.364525 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e102dd21-a348-4c82-8cc1-3938b7e68387-kube-api-access-vdlp7" pod "e102dd21-a348-4c82-8cc1-3938b7e68387" (UID: "e102dd21-a348-4c82-8cc1-3938b7e68387"). InnerVolumeSpecName "kube-api-access-vdlp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:18.365162 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.365139 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ab5c46-8257-4e49-b936-c83eb203b08f-kube-api-access-x8gb4" pod "05ab5c46-8257-4e49-b936-c83eb203b08f" (UID: "05ab5c46-8257-4e49-b936-c83eb203b08f"). InnerVolumeSpecName "kube-api-access-x8gb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:18.365693 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.365656 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b1e735a-5598-41d3-b557-5161b194e7d3-kube-api-access-jpv4g" pod "9b1e735a-5598-41d3-b557-5161b194e7d3" (UID: "9b1e735a-5598-41d3-b557-5161b194e7d3"). InnerVolumeSpecName "kube-api-access-jpv4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:18.366999 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.366977 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2abea4-3ca9-4866-831d-644af9c391b1-kube-api-access-jdntw" pod "3d2abea4-3ca9-4866-831d-644af9c391b1" (UID: "3d2abea4-3ca9-4866-831d-644af9c391b1"). InnerVolumeSpecName "kube-api-access-jdntw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:18.368485 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.368461 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49365a6-eff4-4d06-a968-9a05e25a70c0-util" pod "e49365a6-eff4-4d06-a968-9a05e25a70c0" (UID: "e49365a6-eff4-4d06-a968-9a05e25a70c0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:18.369118 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.369100 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ab5c46-8257-4e49-b936-c83eb203b08f-util" pod "05ab5c46-8257-4e49-b936-c83eb203b08f" (UID: "05ab5c46-8257-4e49-b936-c83eb203b08f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:18.369912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.369891 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2abea4-3ca9-4866-831d-644af9c391b1-util" pod "3d2abea4-3ca9-4866-831d-644af9c391b1" (UID: "3d2abea4-3ca9-4866-831d-644af9c391b1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:18.370477 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.370458 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1e735a-5598-41d3-b557-5161b194e7d3-util" pod "9b1e735a-5598-41d3-b557-5161b194e7d3" (UID: "9b1e735a-5598-41d3-b557-5161b194e7d3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:18.373473 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.373444 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e102dd21-a348-4c82-8cc1-3938b7e68387-util" pod "e102dd21-a348-4c82-8cc1-3938b7e68387" (UID: "e102dd21-a348-4c82-8cc1-3938b7e68387"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:18.460964 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.460941 4385 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05ab5c46-8257-4e49-b936-c83eb203b08f-bundle\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:18.461103 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.460967 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9x7f\" (UniqueName: \"kubernetes.io/projected/e49365a6-eff4-4d06-a968-9a05e25a70c0-kube-api-access-f9x7f\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:18.461103 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.460983 4385 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e102dd21-a348-4c82-8cc1-3938b7e68387-bundle\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:18.461103 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.460997 4385 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b1e735a-5598-41d3-b557-5161b194e7d3-bundle\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:18.461103 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.461009 4385 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e49365a6-eff4-4d06-a968-9a05e25a70c0-util\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:18.461103 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.461020 4385 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d2abea4-3ca9-4866-831d-644af9c391b1-bundle\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:18.461103 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.461031 4385 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d2abea4-3ca9-4866-831d-644af9c391b1-util\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:18.461103 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.461042 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8gb4\" (UniqueName: \"kubernetes.io/projected/05ab5c46-8257-4e49-b936-c83eb203b08f-kube-api-access-x8gb4\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:18.461103 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.461054 4385 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05ab5c46-8257-4e49-b936-c83eb203b08f-util\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:18.461103 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.461066 4385 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e49365a6-eff4-4d06-a968-9a05e25a70c0-bundle\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:18.461103 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.461077 4385 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b1e735a-5598-41d3-b557-5161b194e7d3-util\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:18.461103 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.461088 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdlp7\" (UniqueName: \"kubernetes.io/projected/e102dd21-a348-4c82-8cc1-3938b7e68387-kube-api-access-vdlp7\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:18.461103 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.461099 4385 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e102dd21-a348-4c82-8cc1-3938b7e68387-util\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:18.461487 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.461113 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdntw\" (UniqueName: \"kubernetes.io/projected/3d2abea4-3ca9-4866-831d-644af9c391b1-kube-api-access-jdntw\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:18.461487 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:18.461127 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpv4g\" (UniqueName: \"kubernetes.io/projected/9b1e735a-5598-41d3-b557-5161b194e7d3-kube-api-access-jpv4g\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:19.019594 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.019560 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc" event={"ID":"05ab5c46-8257-4e49-b936-c83eb203b08f","Type":"ContainerDied","Data":"06bdfd9afd40ee08fc85ed43a96b861711a550f2cdc41c1ff19a6e2badb6d420"} Nov 01 06:14:19.019594 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.019581 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644m9vgc" Nov 01 06:14:19.020076 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.019653 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06bdfd9afd40ee08fc85ed43a96b861711a550f2cdc41c1ff19a6e2badb6d420" Nov 01 06:14:19.021696 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.021672 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm" Nov 01 06:14:19.021913 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.021673 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6njflm" event={"ID":"9b1e735a-5598-41d3-b557-5161b194e7d3","Type":"ContainerDied","Data":"17186c25dd41ea2fd7bfba1abb2f98fea82b482c52095ac381c62ea2e6039a27"} Nov 01 06:14:19.021913 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.021906 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17186c25dd41ea2fd7bfba1abb2f98fea82b482c52095ac381c62ea2e6039a27" Nov 01 06:14:19.023884 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.023858 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc" event={"ID":"e102dd21-a348-4c82-8cc1-3938b7e68387","Type":"ContainerDied","Data":"c71567dbaaa2813657311f0d5d72a46045501149267c3244a7c507d026868db7"} Nov 01 06:14:19.023884 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.023877 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sjnrc" Nov 01 06:14:19.024059 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.023898 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c71567dbaaa2813657311f0d5d72a46045501149267c3244a7c507d026868db7" Nov 01 06:14:19.026234 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.026217 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv" Nov 01 06:14:19.026335 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.026213 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4swv" event={"ID":"c591503c-5b74-45c7-a3f5-13cbecb15650","Type":"ContainerDied","Data":"763b74bccf2609173bc2c029c2631481a2a6daec7d6af67f9432489fef777ac1"} Nov 01 06:14:19.026335 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.026327 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="763b74bccf2609173bc2c029c2631481a2a6daec7d6af67f9432489fef777ac1" Nov 01 06:14:19.028376 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.028355 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9" event={"ID":"e49365a6-eff4-4d06-a968-9a05e25a70c0","Type":"ContainerDied","Data":"ad18db0979704ffb5002ccf0f4cd6822ea23e1c3e244a1406974556376e32b41"} Nov 01 06:14:19.028472 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.028393 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad18db0979704ffb5002ccf0f4cd6822ea23e1c3e244a1406974556376e32b41" Nov 01 06:14:19.028472 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.028466 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ankzp9" Nov 01 06:14:19.030294 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.030270 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7" event={"ID":"3d2abea4-3ca9-4866-831d-644af9c391b1","Type":"ContainerDied","Data":"cdc332c0eda8e0536dd9effbed9828480a87febc567150500b96748363cd5c69"} Nov 01 06:14:19.030381 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.030308 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdc332c0eda8e0536dd9effbed9828480a87febc567150500b96748363cd5c69" Nov 01 06:14:19.030381 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.030313 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dd9d7" Nov 01 06:14:19.171070 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.171042 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j" Nov 01 06:14:19.266748 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.266709 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/885a807a-2d27-4f2b-8a04-7a6c1388820b-kube-api-access-4mbkt\" (UniqueName: \"kubernetes.io/projected/885a807a-2d27-4f2b-8a04-7a6c1388820b-kube-api-access-4mbkt\") pod \"885a807a-2d27-4f2b-8a04-7a6c1388820b\" (UID: \"885a807a-2d27-4f2b-8a04-7a6c1388820b\") " Nov 01 06:14:19.266946 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.266792 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/885a807a-2d27-4f2b-8a04-7a6c1388820b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/885a807a-2d27-4f2b-8a04-7a6c1388820b-bundle\") pod \"885a807a-2d27-4f2b-8a04-7a6c1388820b\" (UID: \"885a807a-2d27-4f2b-8a04-7a6c1388820b\") " Nov 01 06:14:19.266946 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.266828 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/885a807a-2d27-4f2b-8a04-7a6c1388820b-util\" (UniqueName: \"kubernetes.io/empty-dir/885a807a-2d27-4f2b-8a04-7a6c1388820b-util\") pod \"885a807a-2d27-4f2b-8a04-7a6c1388820b\" (UID: \"885a807a-2d27-4f2b-8a04-7a6c1388820b\") " Nov 01 06:14:19.267257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.267233 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/885a807a-2d27-4f2b-8a04-7a6c1388820b-bundle" pod "885a807a-2d27-4f2b-8a04-7a6c1388820b" (UID: "885a807a-2d27-4f2b-8a04-7a6c1388820b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:19.269075 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.269051 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/885a807a-2d27-4f2b-8a04-7a6c1388820b-kube-api-access-4mbkt" pod "885a807a-2d27-4f2b-8a04-7a6c1388820b" (UID: "885a807a-2d27-4f2b-8a04-7a6c1388820b"). InnerVolumeSpecName "kube-api-access-4mbkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:14:19.272643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.272587 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/885a807a-2d27-4f2b-8a04-7a6c1388820b-util" pod "885a807a-2d27-4f2b-8a04-7a6c1388820b" (UID: "885a807a-2d27-4f2b-8a04-7a6c1388820b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:14:19.367553 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.367517 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mbkt\" (UniqueName: \"kubernetes.io/projected/885a807a-2d27-4f2b-8a04-7a6c1388820b-kube-api-access-4mbkt\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:19.367553 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.367545 4385 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/885a807a-2d27-4f2b-8a04-7a6c1388820b-bundle\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:19.367553 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:19.367554 4385 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/885a807a-2d27-4f2b-8a04-7a6c1388820b-util\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:14:20.037855 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:20.037811 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j" Nov 01 06:14:20.037855 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:20.037822 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6444dl8j" event={"ID":"885a807a-2d27-4f2b-8a04-7a6c1388820b","Type":"ContainerDied","Data":"1eae191da64eeb6a31e8357cbe2bb98c22c1e39c8fbb83882ad1f2d6ff2b49e5"} Nov 01 06:14:20.038367 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:20.037885 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1eae191da64eeb6a31e8357cbe2bb98c22c1e39c8fbb83882ad1f2d6ff2b49e5" Nov 01 06:14:20.854072 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:20.854019 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:21.704921 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.704890 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g"] Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705166 4385 cpu_manager.go:414] "containerMap: removing container" podUID="9b1e735a-5598-41d3-b557-5161b194e7d3" containerName="pull" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705177 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b1e735a-5598-41d3-b557-5161b194e7d3" containerName="pull" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705187 4385 cpu_manager.go:414] "containerMap: removing container" podUID="05ab5c46-8257-4e49-b936-c83eb203b08f" containerName="util" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705192 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05ab5c46-8257-4e49-b936-c83eb203b08f" containerName="util" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705201 4385 cpu_manager.go:414] "containerMap: removing container" podUID="05ab5c46-8257-4e49-b936-c83eb203b08f" containerName="pull" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705206 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05ab5c46-8257-4e49-b936-c83eb203b08f" containerName="pull" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705213 4385 cpu_manager.go:414] "containerMap: removing container" podUID="e49365a6-eff4-4d06-a968-9a05e25a70c0" containerName="pull" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705218 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e49365a6-eff4-4d06-a968-9a05e25a70c0" containerName="pull" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705225 4385 cpu_manager.go:414] "containerMap: removing container" podUID="c591503c-5b74-45c7-a3f5-13cbecb15650" containerName="pull" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705229 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c591503c-5b74-45c7-a3f5-13cbecb15650" containerName="pull" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705235 4385 cpu_manager.go:414] "containerMap: removing container" podUID="2ea4d43f-2d68-435a-adec-081098268d28" containerName="extract" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705239 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea4d43f-2d68-435a-adec-081098268d28" containerName="extract" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705245 4385 cpu_manager.go:414] "containerMap: removing container" podUID="e102dd21-a348-4c82-8cc1-3938b7e68387" containerName="extract" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705250 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e102dd21-a348-4c82-8cc1-3938b7e68387" containerName="extract" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705255 4385 cpu_manager.go:414] "containerMap: removing container" podUID="9b1e735a-5598-41d3-b557-5161b194e7d3" containerName="extract" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705259 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b1e735a-5598-41d3-b557-5161b194e7d3" containerName="extract" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705265 4385 cpu_manager.go:414] "containerMap: removing container" podUID="e102dd21-a348-4c82-8cc1-3938b7e68387" containerName="util" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705270 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e102dd21-a348-4c82-8cc1-3938b7e68387" containerName="util" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705275 4385 cpu_manager.go:414] "containerMap: removing container" podUID="2ea4d43f-2d68-435a-adec-081098268d28" containerName="pull" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705279 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea4d43f-2d68-435a-adec-081098268d28" containerName="pull" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705288 4385 cpu_manager.go:414] "containerMap: removing container" podUID="77c51a13-98cf-45ab-a049-bac31d7cada0" containerName="util" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705292 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c51a13-98cf-45ab-a049-bac31d7cada0" containerName="util" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705297 4385 cpu_manager.go:414] "containerMap: removing container" podUID="75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8" containerName="pull" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705301 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8" containerName="pull" Nov 01 06:14:21.705293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705306 4385 cpu_manager.go:414] "containerMap: removing container" podUID="77c51a13-98cf-45ab-a049-bac31d7cada0" containerName="pull" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705310 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c51a13-98cf-45ab-a049-bac31d7cada0" containerName="pull" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705316 4385 cpu_manager.go:414] "containerMap: removing container" podUID="b02fadf0-b23a-4006-a0d8-7669d9565d7b" containerName="pull" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705320 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02fadf0-b23a-4006-a0d8-7669d9565d7b" containerName="pull" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705325 4385 cpu_manager.go:414] "containerMap: removing container" podUID="c591503c-5b74-45c7-a3f5-13cbecb15650" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705329 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c591503c-5b74-45c7-a3f5-13cbecb15650" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705335 4385 cpu_manager.go:414] "containerMap: removing container" podUID="885a807a-2d27-4f2b-8a04-7a6c1388820b" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705340 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="885a807a-2d27-4f2b-8a04-7a6c1388820b" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705348 4385 cpu_manager.go:414] "containerMap: removing container" podUID="2ea4d43f-2d68-435a-adec-081098268d28" containerName="util" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705352 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea4d43f-2d68-435a-adec-081098268d28" containerName="util" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705357 4385 cpu_manager.go:414] "containerMap: removing container" podUID="9306df59-d376-411e-8fa4-6606ed054782" containerName="util" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705361 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9306df59-d376-411e-8fa4-6606ed054782" containerName="util" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705369 4385 cpu_manager.go:414] "containerMap: removing container" podUID="885a807a-2d27-4f2b-8a04-7a6c1388820b" containerName="util" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705372 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="885a807a-2d27-4f2b-8a04-7a6c1388820b" containerName="util" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705378 4385 cpu_manager.go:414] "containerMap: removing container" podUID="3d2abea4-3ca9-4866-831d-644af9c391b1" containerName="pull" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705382 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2abea4-3ca9-4866-831d-644af9c391b1" containerName="pull" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705388 4385 cpu_manager.go:414] "containerMap: removing container" podUID="c591503c-5b74-45c7-a3f5-13cbecb15650" containerName="util" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705393 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c591503c-5b74-45c7-a3f5-13cbecb15650" containerName="util" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705399 4385 cpu_manager.go:414] "containerMap: removing container" podUID="e49365a6-eff4-4d06-a968-9a05e25a70c0" containerName="util" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705403 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e49365a6-eff4-4d06-a968-9a05e25a70c0" containerName="util" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705408 4385 cpu_manager.go:414] "containerMap: removing container" podUID="9306df59-d376-411e-8fa4-6606ed054782" containerName="pull" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705412 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9306df59-d376-411e-8fa4-6606ed054782" containerName="pull" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705418 4385 cpu_manager.go:414] "containerMap: removing container" podUID="9306df59-d376-411e-8fa4-6606ed054782" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705423 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9306df59-d376-411e-8fa4-6606ed054782" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705428 4385 cpu_manager.go:414] "containerMap: removing container" podUID="3d2abea4-3ca9-4866-831d-644af9c391b1" containerName="util" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705432 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2abea4-3ca9-4866-831d-644af9c391b1" containerName="util" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705438 4385 cpu_manager.go:414] "containerMap: removing container" podUID="9b1e735a-5598-41d3-b557-5161b194e7d3" containerName="util" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705442 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b1e735a-5598-41d3-b557-5161b194e7d3" containerName="util" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705447 4385 cpu_manager.go:414] "containerMap: removing container" podUID="b02fadf0-b23a-4006-a0d8-7669d9565d7b" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705451 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02fadf0-b23a-4006-a0d8-7669d9565d7b" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705458 4385 cpu_manager.go:414] "containerMap: removing container" podUID="75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705462 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705470 4385 cpu_manager.go:414] "containerMap: removing container" podUID="75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8" containerName="util" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705475 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8" containerName="util" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705481 4385 cpu_manager.go:414] "containerMap: removing container" podUID="05ab5c46-8257-4e49-b936-c83eb203b08f" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705485 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05ab5c46-8257-4e49-b936-c83eb203b08f" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705491 4385 cpu_manager.go:414] "containerMap: removing container" podUID="3d2abea4-3ca9-4866-831d-644af9c391b1" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705496 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2abea4-3ca9-4866-831d-644af9c391b1" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705502 4385 cpu_manager.go:414] "containerMap: removing container" podUID="b02fadf0-b23a-4006-a0d8-7669d9565d7b" containerName="util" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705506 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02fadf0-b23a-4006-a0d8-7669d9565d7b" containerName="util" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705512 4385 cpu_manager.go:414] "containerMap: removing container" podUID="77c51a13-98cf-45ab-a049-bac31d7cada0" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705517 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c51a13-98cf-45ab-a049-bac31d7cada0" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705523 4385 cpu_manager.go:414] "containerMap: removing container" podUID="885a807a-2d27-4f2b-8a04-7a6c1388820b" containerName="pull" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705527 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="885a807a-2d27-4f2b-8a04-7a6c1388820b" containerName="pull" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705532 4385 cpu_manager.go:414] "containerMap: removing container" podUID="e102dd21-a348-4c82-8cc1-3938b7e68387" containerName="pull" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705536 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e102dd21-a348-4c82-8cc1-3938b7e68387" containerName="pull" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705542 4385 cpu_manager.go:414] "containerMap: removing container" podUID="e49365a6-eff4-4d06-a968-9a05e25a70c0" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705546 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e49365a6-eff4-4d06-a968-9a05e25a70c0" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705585 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="05ab5c46-8257-4e49-b936-c83eb203b08f" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705593 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2abea4-3ca9-4866-831d-644af9c391b1" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705599 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c51a13-98cf-45ab-a049-bac31d7cada0" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705605 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="e49365a6-eff4-4d06-a968-9a05e25a70c0" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705611 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b1e735a-5598-41d3-b557-5161b194e7d3" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705616 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea4d43f-2d68-435a-adec-081098268d28" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705621 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="e102dd21-a348-4c82-8cc1-3938b7e68387" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705629 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="c591503c-5b74-45c7-a3f5-13cbecb15650" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705634 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="885a807a-2d27-4f2b-8a04-7a6c1388820b" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705640 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="9306df59-d376-411e-8fa4-6606ed054782" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705644 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02fadf0-b23a-4006-a0d8-7669d9565d7b" containerName="extract" Nov 01 06:14:21.706013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.705650 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f7ef2f-8d4b-4d84-aab9-1bbdb7c370b8" containerName="extract" Nov 01 06:14:21.707563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.707546 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:21.760339 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.760304 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 06:14:21.760540 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.760349 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 06:14:21.760540 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.760478 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 06:14:21.760673 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.760540 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 06:14:21.760673 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.760543 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 06:14:21.760673 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.760587 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-qjjpq\"" Nov 01 06:14:21.789371 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.789337 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5e919603-8c25-49c5-9dae-0af5ed7a4362-manifests\") pod \"opendatahub-operator-controller-manager-5965b9bff7-zmv2g\" (UID: \"5e919603-8c25-49c5-9dae-0af5ed7a4362\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:21.789532 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.789395 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5e919603-8c25-49c5-9dae-0af5ed7a4362-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-zmv2g\" (UID: \"5e919603-8c25-49c5-9dae-0af5ed7a4362\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:21.789532 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.789450 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e919603-8c25-49c5-9dae-0af5ed7a4362-tmp\") pod \"opendatahub-operator-controller-manager-5965b9bff7-zmv2g\" (UID: \"5e919603-8c25-49c5-9dae-0af5ed7a4362\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:21.789532 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.789491 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e919603-8c25-49c5-9dae-0af5ed7a4362-webhook-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-zmv2g\" (UID: \"5e919603-8c25-49c5-9dae-0af5ed7a4362\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:21.789532 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.789513 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pc92\" (UniqueName: \"kubernetes.io/projected/5e919603-8c25-49c5-9dae-0af5ed7a4362-kube-api-access-2pc92\") pod \"opendatahub-operator-controller-manager-5965b9bff7-zmv2g\" (UID: \"5e919603-8c25-49c5-9dae-0af5ed7a4362\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:21.789672 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.789609 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e919603-8c25-49c5-9dae-0af5ed7a4362-metrics-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-zmv2g\" (UID: \"5e919603-8c25-49c5-9dae-0af5ed7a4362\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:21.814933 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.814896 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g"] Nov 01 06:14:21.889987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.889948 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5e919603-8c25-49c5-9dae-0af5ed7a4362-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-zmv2g\" (UID: \"5e919603-8c25-49c5-9dae-0af5ed7a4362\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:21.889987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.889985 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e919603-8c25-49c5-9dae-0af5ed7a4362-tmp\") pod \"opendatahub-operator-controller-manager-5965b9bff7-zmv2g\" (UID: \"5e919603-8c25-49c5-9dae-0af5ed7a4362\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:21.890203 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.890005 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e919603-8c25-49c5-9dae-0af5ed7a4362-webhook-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-zmv2g\" (UID: \"5e919603-8c25-49c5-9dae-0af5ed7a4362\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:21.890203 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.890123 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pc92\" (UniqueName: \"kubernetes.io/projected/5e919603-8c25-49c5-9dae-0af5ed7a4362-kube-api-access-2pc92\") pod \"opendatahub-operator-controller-manager-5965b9bff7-zmv2g\" (UID: \"5e919603-8c25-49c5-9dae-0af5ed7a4362\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:21.890203 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.890198 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e919603-8c25-49c5-9dae-0af5ed7a4362-metrics-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-zmv2g\" (UID: \"5e919603-8c25-49c5-9dae-0af5ed7a4362\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:21.890373 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.890222 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5e919603-8c25-49c5-9dae-0af5ed7a4362-manifests\") pod \"opendatahub-operator-controller-manager-5965b9bff7-zmv2g\" (UID: \"5e919603-8c25-49c5-9dae-0af5ed7a4362\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:21.890583 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.890562 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5e919603-8c25-49c5-9dae-0af5ed7a4362-manifests\") pod \"opendatahub-operator-controller-manager-5965b9bff7-zmv2g\" (UID: \"5e919603-8c25-49c5-9dae-0af5ed7a4362\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:21.892334 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.892306 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e919603-8c25-49c5-9dae-0af5ed7a4362-tmp\") pod \"opendatahub-operator-controller-manager-5965b9bff7-zmv2g\" (UID: \"5e919603-8c25-49c5-9dae-0af5ed7a4362\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:21.892599 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.892579 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5e919603-8c25-49c5-9dae-0af5ed7a4362-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-zmv2g\" (UID: \"5e919603-8c25-49c5-9dae-0af5ed7a4362\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:21.892686 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.892610 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e919603-8c25-49c5-9dae-0af5ed7a4362-metrics-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-zmv2g\" (UID: \"5e919603-8c25-49c5-9dae-0af5ed7a4362\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:21.892736 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.892692 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e919603-8c25-49c5-9dae-0af5ed7a4362-webhook-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-zmv2g\" (UID: \"5e919603-8c25-49c5-9dae-0af5ed7a4362\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:21.946980 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:21.946945 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pc92\" (UniqueName: \"kubernetes.io/projected/5e919603-8c25-49c5-9dae-0af5ed7a4362-kube-api-access-2pc92\") pod \"opendatahub-operator-controller-manager-5965b9bff7-zmv2g\" (UID: \"5e919603-8c25-49c5-9dae-0af5ed7a4362\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:22.019758 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:22.019663 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:22.499870 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:22.499821 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g"] Nov 01 06:14:22.500731 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:14:22.500703 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e919603_8c25_49c5_9dae_0af5ed7a4362.slice/crio-6ed5b3e640dc9cb6ce5643fcc3757dbb58e1c7d5a207a303a4485f6328cd0f3d WatchSource:0}: Error finding container 6ed5b3e640dc9cb6ce5643fcc3757dbb58e1c7d5a207a303a4485f6328cd0f3d: Status 404 returned error can't find the container with id 6ed5b3e640dc9cb6ce5643fcc3757dbb58e1c7d5a207a303a4485f6328cd0f3d Nov 01 06:14:22.528770 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:22.528742 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:23.053892 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:23.053852 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" event={"ID":"5e919603-8c25-49c5-9dae-0af5ed7a4362","Type":"ContainerStarted","Data":"6ed5b3e640dc9cb6ce5643fcc3757dbb58e1c7d5a207a303a4485f6328cd0f3d"} Nov 01 06:14:25.277584 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:25.277540 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:25.861474 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:25.861430 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:26.072579 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:26.072545 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" event={"ID":"5e919603-8c25-49c5-9dae-0af5ed7a4362","Type":"ContainerStarted","Data":"431111f5619a381d21e61d02a3ec33220d190396623be2f442d4007bb8010802"} Nov 01 06:14:27.079014 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:27.078978 4385 generic.go:356] "Generic (PLEG): container finished" podID="5e919603-8c25-49c5-9dae-0af5ed7a4362" containerID="431111f5619a381d21e61d02a3ec33220d190396623be2f442d4007bb8010802" exitCode=0 Nov 01 06:14:27.079478 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:27.079026 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" event={"ID":"5e919603-8c25-49c5-9dae-0af5ed7a4362","Type":"ContainerDied","Data":"431111f5619a381d21e61d02a3ec33220d190396623be2f442d4007bb8010802"} Nov 01 06:14:27.438200 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:27.438111 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:28.084866 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:28.084809 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" event={"ID":"5e919603-8c25-49c5-9dae-0af5ed7a4362","Type":"ContainerStarted","Data":"98881578286ab5436a1cfa8b5fbc7d81cba00224ec5aa2a224eb28bec2897161"} Nov 01 06:14:28.085331 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:28.084943 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:28.261542 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:28.261432 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" podStartSLOduration=2.597760477 podStartE2EDuration="7.261417819s" totalImagesPullingTime="4.219048919s" totalInitContainerRuntime="444.608423ms" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:14:28.260111677 +0000 UTC m=+278.023911533" watchObservedRunningTime="2026-11-01 06:14:28.261417819 +0000 UTC m=+278.025217675" Nov 01 06:14:28.867443 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:28.867401 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:29.031003 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:29.030964 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:29.473191 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:29.473160 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7mrd"] Nov 01 06:14:29.475218 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:29.475204 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7mrd" Nov 01 06:14:29.528067 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:29.528041 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 06:14:29.577758 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:29.577723 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 06:14:29.577959 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:29.577789 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5j7fl\"" Nov 01 06:14:29.580189 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:29.580166 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7mrd"] Nov 01 06:14:29.658765 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:29.658718 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl9mp\" (UniqueName: \"kubernetes.io/projected/79830bd7-36da-4336-86d4-8e63d1a7e440-kube-api-access-rl9mp\") pod \"limitador-operator-controller-manager-5778bb4449-q7mrd\" (UID: \"79830bd7-36da-4336-86d4-8e63d1a7e440\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7mrd" Nov 01 06:14:29.759715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:29.759613 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl9mp\" (UniqueName: \"kubernetes.io/projected/79830bd7-36da-4336-86d4-8e63d1a7e440-kube-api-access-rl9mp\") pod \"limitador-operator-controller-manager-5778bb4449-q7mrd\" (UID: \"79830bd7-36da-4336-86d4-8e63d1a7e440\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7mrd" Nov 01 06:14:29.816336 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:29.816294 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl9mp\" (UniqueName: \"kubernetes.io/projected/79830bd7-36da-4336-86d4-8e63d1a7e440-kube-api-access-rl9mp\") pod \"limitador-operator-controller-manager-5778bb4449-q7mrd\" (UID: \"79830bd7-36da-4336-86d4-8e63d1a7e440\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7mrd" Nov 01 06:14:30.087240 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:30.087190 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7mrd" Nov 01 06:14:30.461809 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:30.461786 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7mrd"] Nov 01 06:14:30.464092 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:14:30.464063 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79830bd7_36da_4336_86d4_8e63d1a7e440.slice/crio-cad7c0b824efa656368dd0728482422541708a3f96c4c87d0ea8ca97d7281bb9 WatchSource:0}: Error finding container cad7c0b824efa656368dd0728482422541708a3f96c4c87d0ea8ca97d7281bb9: Status 404 returned error can't find the container with id cad7c0b824efa656368dd0728482422541708a3f96c4c87d0ea8ca97d7281bb9 Nov 01 06:14:31.099298 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:31.099263 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7mrd" event={"ID":"79830bd7-36da-4336-86d4-8e63d1a7e440","Type":"ContainerStarted","Data":"cad7c0b824efa656368dd0728482422541708a3f96c4c87d0ea8ca97d7281bb9"} Nov 01 06:14:31.257367 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:31.257307 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:33.925435 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:33.925404 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-5hrk6"] Nov 01 06:14:33.928698 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:33.928673 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5hrk6" Nov 01 06:14:34.030621 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:34.030578 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 01 06:14:34.030621 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:34.030585 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-gqgvs\"" Nov 01 06:14:34.032918 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:34.032891 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-5hrk6"] Nov 01 06:14:34.097803 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:34.097767 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vljcq\" (UniqueName: \"kubernetes.io/projected/d6a699b6-b7e6-4e66-8a91-2db8f6209e11-kube-api-access-vljcq\") pod \"dns-operator-controller-manager-8557cbdb84-5hrk6\" (UID: \"d6a699b6-b7e6-4e66-8a91-2db8f6209e11\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5hrk6" Nov 01 06:14:34.115385 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:34.115356 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7mrd" event={"ID":"79830bd7-36da-4336-86d4-8e63d1a7e440","Type":"ContainerStarted","Data":"79732012d47b16fff79f340bfee4788a9800e1486b7e59ef7612e09c673161fd"} Nov 01 06:14:34.115505 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:34.115422 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7mrd" Nov 01 06:14:34.199020 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:34.198931 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vljcq\" (UniqueName: \"kubernetes.io/projected/d6a699b6-b7e6-4e66-8a91-2db8f6209e11-kube-api-access-vljcq\") pod \"dns-operator-controller-manager-8557cbdb84-5hrk6\" (UID: \"d6a699b6-b7e6-4e66-8a91-2db8f6209e11\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5hrk6" Nov 01 06:14:34.230678 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:34.230627 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7mrd" podStartSLOduration=1.7337322579999999 podStartE2EDuration="5.230611753s" totalImagesPullingTime="3.496879495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:14:34.22897298 +0000 UTC m=+283.992772836" watchObservedRunningTime="2026-11-01 06:14:34.230611753 +0000 UTC m=+283.994411609" Nov 01 06:14:34.255615 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:34.255576 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vljcq\" (UniqueName: \"kubernetes.io/projected/d6a699b6-b7e6-4e66-8a91-2db8f6209e11-kube-api-access-vljcq\") pod \"dns-operator-controller-manager-8557cbdb84-5hrk6\" (UID: \"d6a699b6-b7e6-4e66-8a91-2db8f6209e11\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5hrk6" Nov 01 06:14:34.542068 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:34.541978 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5hrk6" Nov 01 06:14:34.675074 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:34.675037 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:34.915656 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:34.915631 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-5hrk6"] Nov 01 06:14:34.917520 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:14:34.917495 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a699b6_b7e6_4e66_8a91_2db8f6209e11.slice/crio-8255fcfed604259b20ac17fd080300a708d5c01d2e1d4987d19a3d9cfbe67eeb WatchSource:0}: Error finding container 8255fcfed604259b20ac17fd080300a708d5c01d2e1d4987d19a3d9cfbe67eeb: Status 404 returned error can't find the container with id 8255fcfed604259b20ac17fd080300a708d5c01d2e1d4987d19a3d9cfbe67eeb Nov 01 06:14:35.121371 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:35.121323 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5hrk6" event={"ID":"d6a699b6-b7e6-4e66-8a91-2db8f6209e11","Type":"ContainerStarted","Data":"8255fcfed604259b20ac17fd080300a708d5c01d2e1d4987d19a3d9cfbe67eeb"} Nov 01 06:14:35.583339 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:35.583291 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:35.836075 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:35.835980 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:36.246134 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:36.246033 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:36.655137 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:36.655044 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:36.779824 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:14:36.779790 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:14:38.136204 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:38.136168 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5hrk6" event={"ID":"d6a699b6-b7e6-4e66-8a91-2db8f6209e11","Type":"ContainerStarted","Data":"1088e02abbaa3c5db443de4270a6236caff8febdcce2037f9d6b604821aac77d"} Nov 01 06:14:38.136640 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:38.136229 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5hrk6" Nov 01 06:14:38.250753 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:38.250698 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5hrk6" podStartSLOduration=2.729882736 podStartE2EDuration="5.250682534s" totalImagesPullingTime="2.520799798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:14:38.249127124 +0000 UTC m=+288.012926981" watchObservedRunningTime="2026-11-01 06:14:38.250682534 +0000 UTC m=+288.014482391" Nov 01 06:14:39.091689 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:39.091660 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-zmv2g" Nov 01 06:14:39.854967 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:39.854931 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5dm8z"] Nov 01 06:14:39.878227 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:39.878194 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5dm8z" Nov 01 06:14:39.918052 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:39.918021 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5dm8z"] Nov 01 06:14:39.930332 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:39.930305 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 01 06:14:39.930332 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:39.930305 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 01 06:14:39.930332 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:39.930305 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-b7gdh\"" Nov 01 06:14:39.940820 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:39.940790 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0723bd9f-cfcb-4d93-bda5-c945056c2368-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-5dm8z\" (UID: \"0723bd9f-cfcb-4d93-bda5-c945056c2368\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5dm8z" Nov 01 06:14:39.940944 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:39.940927 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5gmt\" (UniqueName: \"kubernetes.io/projected/0723bd9f-cfcb-4d93-bda5-c945056c2368-kube-api-access-d5gmt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-5dm8z\" (UID: \"0723bd9f-cfcb-4d93-bda5-c945056c2368\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5dm8z" Nov 01 06:14:40.041537 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:40.041507 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0723bd9f-cfcb-4d93-bda5-c945056c2368-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-5dm8z\" (UID: \"0723bd9f-cfcb-4d93-bda5-c945056c2368\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5dm8z" Nov 01 06:14:40.041688 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:40.041562 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5gmt\" (UniqueName: \"kubernetes.io/projected/0723bd9f-cfcb-4d93-bda5-c945056c2368-kube-api-access-d5gmt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-5dm8z\" (UID: \"0723bd9f-cfcb-4d93-bda5-c945056c2368\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5dm8z" Nov 01 06:14:40.041933 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:40.041916 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0723bd9f-cfcb-4d93-bda5-c945056c2368-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-5dm8z\" (UID: \"0723bd9f-cfcb-4d93-bda5-c945056c2368\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5dm8z" Nov 01 06:14:40.102764 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:40.102729 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5gmt\" (UniqueName: \"kubernetes.io/projected/0723bd9f-cfcb-4d93-bda5-c945056c2368-kube-api-access-d5gmt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-5dm8z\" (UID: \"0723bd9f-cfcb-4d93-bda5-c945056c2368\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5dm8z" Nov 01 06:14:40.204940 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:40.204818 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5dm8z" Nov 01 06:14:40.651989 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:40.651959 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5dm8z"] Nov 01 06:14:40.879896 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:40.879868 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:41.151249 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:41.151214 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5dm8z" event={"ID":"0723bd9f-cfcb-4d93-bda5-c945056c2368","Type":"ContainerStarted","Data":"3c3b45a206fec2a20eb2784195bece3633b3283ff0e6385c444a2c42002ba89b"} Nov 01 06:14:41.156143 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:41.156116 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:42.000349 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:42.000316 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg"] Nov 01 06:14:42.003196 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:42.003179 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg" Nov 01 06:14:42.060254 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:42.060219 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-99hfg\" (UID: \"7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg" Nov 01 06:14:42.060427 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:42.060273 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnxsp\" (UniqueName: \"kubernetes.io/projected/7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f-kube-api-access-pnxsp\") pod \"kuadrant-operator-controller-manager-cf8469d7b-99hfg\" (UID: \"7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg" Nov 01 06:14:42.111539 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:42.111501 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rbmtd\"" Nov 01 06:14:42.113768 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:42.113741 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg"] Nov 01 06:14:42.161236 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:42.161200 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-99hfg\" (UID: \"7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg" Nov 01 06:14:42.161405 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:42.161297 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnxsp\" (UniqueName: \"kubernetes.io/projected/7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f-kube-api-access-pnxsp\") pod \"kuadrant-operator-controller-manager-cf8469d7b-99hfg\" (UID: \"7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg" Nov 01 06:14:42.161602 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:42.161578 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-99hfg\" (UID: \"7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg" Nov 01 06:14:42.221614 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:42.221579 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnxsp\" (UniqueName: \"kubernetes.io/projected/7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f-kube-api-access-pnxsp\") pod \"kuadrant-operator-controller-manager-cf8469d7b-99hfg\" (UID: \"7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg" Nov 01 06:14:42.254583 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:42.254488 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:42.314477 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:42.314441 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg" Nov 01 06:14:42.481794 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:42.481757 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:42.642225 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:42.642191 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:42.748921 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:42.748886 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg"] Nov 01 06:14:42.750935 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:14:42.750898 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab5f745_99de_4cb5_8c8d_c0a33a3ad88f.slice/crio-acb7e9458c15108c4d8f12f5cbfd6a7ef0e8b82a024b95bc57bb9f72137d69d8 WatchSource:0}: Error finding container acb7e9458c15108c4d8f12f5cbfd6a7ef0e8b82a024b95bc57bb9f72137d69d8: Status 404 returned error can't find the container with id acb7e9458c15108c4d8f12f5cbfd6a7ef0e8b82a024b95bc57bb9f72137d69d8 Nov 01 06:14:43.125006 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:43.124967 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:43.162639 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:43.162589 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5dm8z" event={"ID":"0723bd9f-cfcb-4d93-bda5-c945056c2368","Type":"ContainerStarted","Data":"c5c2073acbabbcecfc01e1e5ee23493c291e12dce0a4e1ca1f97ca21910924e4"} Nov 01 06:14:43.164362 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:43.164337 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg" event={"ID":"7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f","Type":"ContainerStarted","Data":"acb7e9458c15108c4d8f12f5cbfd6a7ef0e8b82a024b95bc57bb9f72137d69d8"} Nov 01 06:14:43.321974 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:43.321924 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5dm8z" podStartSLOduration=1.877858888 podStartE2EDuration="4.321910797s" totalImagesPullingTime="2.444051909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:14:43.319865388 +0000 UTC m=+293.083665237" watchObservedRunningTime="2026-11-01 06:14:43.321910797 +0000 UTC m=+293.085710654" Nov 01 06:14:43.606161 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:43.606130 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:44.946131 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:44.946095 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:45.124220 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:45.124193 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7mrd" Nov 01 06:14:45.222290 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:45.222161 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:45.381852 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:45.381808 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:46.413374 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:46.413327 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-6jbbb"] Nov 01 06:14:46.415905 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:46.415881 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6jbbb" Nov 01 06:14:46.473552 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:46.473515 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-fv9kx\"" Nov 01 06:14:46.493022 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:46.492988 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bhvj\" (UniqueName: \"kubernetes.io/projected/1203942c-3f66-4dbc-b88a-7ab1ea9154be-kube-api-access-7bhvj\") pod \"authorino-operator-7b7bb4bc77-6jbbb\" (UID: \"1203942c-3f66-4dbc-b88a-7ab1ea9154be\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-6jbbb" Nov 01 06:14:46.523149 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:46.523106 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-6jbbb"] Nov 01 06:14:46.594182 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:46.594145 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bhvj\" (UniqueName: \"kubernetes.io/projected/1203942c-3f66-4dbc-b88a-7ab1ea9154be-kube-api-access-7bhvj\") pod \"authorino-operator-7b7bb4bc77-6jbbb\" (UID: \"1203942c-3f66-4dbc-b88a-7ab1ea9154be\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-6jbbb" Nov 01 06:14:46.652305 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:46.652260 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bhvj\" (UniqueName: \"kubernetes.io/projected/1203942c-3f66-4dbc-b88a-7ab1ea9154be-kube-api-access-7bhvj\") pod \"authorino-operator-7b7bb4bc77-6jbbb\" (UID: \"1203942c-3f66-4dbc-b88a-7ab1ea9154be\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-6jbbb" Nov 01 06:14:46.730077 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:46.729997 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6jbbb" Nov 01 06:14:46.778879 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:46.777530 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:47.112697 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:47.112663 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-6jbbb"] Nov 01 06:14:47.116368 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:14:47.116329 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1203942c_3f66_4dbc_b88a_7ab1ea9154be.slice/crio-52e0566b2df9b1a6daf4fcee5515cba779539b1952b7aa72d26833a01bf2949c WatchSource:0}: Error finding container 52e0566b2df9b1a6daf4fcee5515cba779539b1952b7aa72d26833a01bf2949c: Status 404 returned error can't find the container with id 52e0566b2df9b1a6daf4fcee5515cba779539b1952b7aa72d26833a01bf2949c Nov 01 06:14:47.186015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:47.185981 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6jbbb" event={"ID":"1203942c-3f66-4dbc-b88a-7ab1ea9154be","Type":"ContainerStarted","Data":"52e0566b2df9b1a6daf4fcee5515cba779539b1952b7aa72d26833a01bf2949c"} Nov 01 06:14:47.665046 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:47.665007 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:48.193919 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:48.193868 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg" event={"ID":"7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f","Type":"ContainerStarted","Data":"0fc02ebb47f39da93ee79d1d57c52bc08ecb48120b5aa1b0142dcda2d43d0fa9"} Nov 01 06:14:48.194130 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:48.193967 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg" Nov 01 06:14:48.425525 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:48.425467 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg" podStartSLOduration=2.962627635 podStartE2EDuration="7.425450543s" totalImagesPullingTime="4.462822908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:14:48.37459239 +0000 UTC m=+298.138392246" watchObservedRunningTime="2026-11-01 06:14:48.425450543 +0000 UTC m=+298.189250400" Nov 01 06:14:48.858426 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:48.858392 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:49.004318 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:49.004284 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:49.144360 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:49.144282 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5hrk6" Nov 01 06:14:50.205914 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:50.205866 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6jbbb" event={"ID":"1203942c-3f66-4dbc-b88a-7ab1ea9154be","Type":"ContainerStarted","Data":"89b440342d87a15c9d7e4982981848996c16996982d972ed5c272bc95d4906d3"} Nov 01 06:14:50.206349 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:50.205942 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6jbbb" Nov 01 06:14:50.326768 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:50.326695 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6jbbb" podStartSLOduration=2.151568057 podStartE2EDuration="4.326676946s" totalImagesPullingTime="2.175108889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:14:50.322397101 +0000 UTC m=+300.086196957" watchObservedRunningTime="2026-11-01 06:14:50.326676946 +0000 UTC m=+300.090476804" Nov 01 06:14:51.347802 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:51.347761 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:51.521382 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:51.521349 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:51.741591 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:51.741498 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:52.038062 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:52.037974 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:53.053955 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:53.053917 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:53.258979 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:53.258944 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:53.441490 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:53.441452 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:53.603353 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:53.603318 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:53.956428 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:53.956397 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd"] Nov 01 06:14:53.961053 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:53.961026 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:53.996206 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:53.996164 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:54.014260 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.014228 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-rzwv2\"" Nov 01 06:14:54.066215 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.066183 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd"] Nov 01 06:14:54.145309 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.145269 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smzlv\" (UniqueName: \"kubernetes.io/projected/59e2f40b-4458-49f6-a39a-93ad994e8aff-kube-api-access-smzlv\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.145504 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.145323 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59e2f40b-4458-49f6-a39a-93ad994e8aff-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.145504 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.145357 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e2f40b-4458-49f6-a39a-93ad994e8aff-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.145612 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.145525 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59e2f40b-4458-49f6-a39a-93ad994e8aff-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.145612 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.145559 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59e2f40b-4458-49f6-a39a-93ad994e8aff-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.145712 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.145640 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59e2f40b-4458-49f6-a39a-93ad994e8aff-istio-data\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.145712 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.145674 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59e2f40b-4458-49f6-a39a-93ad994e8aff-istio-token\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.145802 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.145744 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e2f40b-4458-49f6-a39a-93ad994e8aff-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.145802 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.145768 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59e2f40b-4458-49f6-a39a-93ad994e8aff-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.243230 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.243155 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:54.246137 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.246115 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e2f40b-4458-49f6-a39a-93ad994e8aff-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.246202 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.246149 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59e2f40b-4458-49f6-a39a-93ad994e8aff-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.246239 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.246201 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smzlv\" (UniqueName: \"kubernetes.io/projected/59e2f40b-4458-49f6-a39a-93ad994e8aff-kube-api-access-smzlv\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.246273 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.246239 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59e2f40b-4458-49f6-a39a-93ad994e8aff-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.246314 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.246289 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e2f40b-4458-49f6-a39a-93ad994e8aff-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.246357 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.246317 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59e2f40b-4458-49f6-a39a-93ad994e8aff-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.246357 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.246340 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59e2f40b-4458-49f6-a39a-93ad994e8aff-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.246470 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.246442 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59e2f40b-4458-49f6-a39a-93ad994e8aff-istio-data\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.246569 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.246552 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59e2f40b-4458-49f6-a39a-93ad994e8aff-istio-token\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.246633 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.246616 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e2f40b-4458-49f6-a39a-93ad994e8aff-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.246683 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.246648 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59e2f40b-4458-49f6-a39a-93ad994e8aff-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.246765 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.246745 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e2f40b-4458-49f6-a39a-93ad994e8aff-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.246823 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.246806 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59e2f40b-4458-49f6-a39a-93ad994e8aff-istio-data\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.247455 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.247427 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59e2f40b-4458-49f6-a39a-93ad994e8aff-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.248856 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.248802 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59e2f40b-4458-49f6-a39a-93ad994e8aff-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.249053 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.249036 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59e2f40b-4458-49f6-a39a-93ad994e8aff-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.305130 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.305099 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59e2f40b-4458-49f6-a39a-93ad994e8aff-istio-token\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.305284 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.305166 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smzlv\" (UniqueName: \"kubernetes.io/projected/59e2f40b-4458-49f6-a39a-93ad994e8aff-kube-api-access-smzlv\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd\" (UID: \"59e2f40b-4458-49f6-a39a-93ad994e8aff\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:54.403382 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.403348 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:54.573918 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:54.573880 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:55.005971 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.005951 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd"] Nov 01 06:14:55.007168 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:14:55.007141 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e2f40b_4458_49f6_a39a_93ad994e8aff.slice/crio-b18e700f167900821c5e500244afcd953ec48ed19ba5d91a4fb3452914f96724 WatchSource:0}: Error finding container b18e700f167900821c5e500244afcd953ec48ed19ba5d91a4fb3452914f96724: Status 404 returned error can't find the container with id b18e700f167900821c5e500244afcd953ec48ed19ba5d91a4fb3452914f96724 Nov 01 06:14:55.009221 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.009185 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:14:55.009313 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.009244 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:14:55.009313 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.009274 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:14:55.231803 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.231771 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" event={"ID":"59e2f40b-4458-49f6-a39a-93ad994e8aff","Type":"ContainerStarted","Data":"7df0fa1b6f35cde970f993b05046568c18905ecf0db08a2f0fcdae055d538251"} Nov 01 06:14:55.232229 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.231825 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" event={"ID":"59e2f40b-4458-49f6-a39a-93ad994e8aff","Type":"ContainerStarted","Data":"b18e700f167900821c5e500244afcd953ec48ed19ba5d91a4fb3452914f96724"} Nov 01 06:14:55.349666 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.349611 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" podStartSLOduration=2.3495969629999998 podStartE2EDuration="2.349596963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:14:55.348591008 +0000 UTC m=+305.112390869" watchObservedRunningTime="2026-11-01 06:14:55.349596963 +0000 UTC m=+305.113396820" Nov 01 06:14:55.518075 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.518044 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwpvc"] Nov 01 06:14:55.519726 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.519698 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:55.520634 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.520621 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwpvc" Nov 01 06:14:55.554017 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.553984 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/045ea95f-c9de-4c1e-af56-ff3712203d6a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pwpvc\" (UID: \"045ea95f-c9de-4c1e-af56-ff3712203d6a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwpvc" Nov 01 06:14:55.554181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.554026 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/045ea95f-c9de-4c1e-af56-ff3712203d6a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pwpvc\" (UID: \"045ea95f-c9de-4c1e-af56-ff3712203d6a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwpvc" Nov 01 06:14:55.554241 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.554170 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqhpz\" (UniqueName: \"kubernetes.io/projected/045ea95f-c9de-4c1e-af56-ff3712203d6a-kube-api-access-zqhpz\") pod \"kuadrant-console-plugin-66c4fb575d-pwpvc\" (UID: \"045ea95f-c9de-4c1e-af56-ff3712203d6a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwpvc" Nov 01 06:14:55.574012 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.573985 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:55.577267 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.577240 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 01 06:14:55.580126 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.580103 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwpvc"] Nov 01 06:14:55.595883 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.595826 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 01 06:14:55.596054 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.596032 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-s9fd7\"" Nov 01 06:14:55.655039 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.654964 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/045ea95f-c9de-4c1e-af56-ff3712203d6a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pwpvc\" (UID: \"045ea95f-c9de-4c1e-af56-ff3712203d6a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwpvc" Nov 01 06:14:55.655039 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.655011 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/045ea95f-c9de-4c1e-af56-ff3712203d6a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pwpvc\" (UID: \"045ea95f-c9de-4c1e-af56-ff3712203d6a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwpvc" Nov 01 06:14:55.655238 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.655079 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqhpz\" (UniqueName: \"kubernetes.io/projected/045ea95f-c9de-4c1e-af56-ff3712203d6a-kube-api-access-zqhpz\") pod \"kuadrant-console-plugin-66c4fb575d-pwpvc\" (UID: \"045ea95f-c9de-4c1e-af56-ff3712203d6a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwpvc" Nov 01 06:14:55.655754 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.655719 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/045ea95f-c9de-4c1e-af56-ff3712203d6a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pwpvc\" (UID: \"045ea95f-c9de-4c1e-af56-ff3712203d6a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwpvc" Nov 01 06:14:55.657556 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.657497 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/045ea95f-c9de-4c1e-af56-ff3712203d6a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pwpvc\" (UID: \"045ea95f-c9de-4c1e-af56-ff3712203d6a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwpvc" Nov 01 06:14:55.716055 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.716020 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqhpz\" (UniqueName: \"kubernetes.io/projected/045ea95f-c9de-4c1e-af56-ff3712203d6a-kube-api-access-zqhpz\") pod \"kuadrant-console-plugin-66c4fb575d-pwpvc\" (UID: \"045ea95f-c9de-4c1e-af56-ff3712203d6a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwpvc" Nov 01 06:14:55.745121 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.745090 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:55.842183 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.842150 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwpvc" Nov 01 06:14:55.919283 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:55.919208 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:56.174613 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:56.174527 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:56.214550 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:56.214528 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwpvc"] Nov 01 06:14:56.216024 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:14:56.215995 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod045ea95f_c9de_4c1e_af56_ff3712203d6a.slice/crio-9dbaeb7ccbb80ff9325a812dcd77bf7aaa3b59874d4009586514ab96e9a05f78 WatchSource:0}: Error finding container 9dbaeb7ccbb80ff9325a812dcd77bf7aaa3b59874d4009586514ab96e9a05f78: Status 404 returned error can't find the container with id 9dbaeb7ccbb80ff9325a812dcd77bf7aaa3b59874d4009586514ab96e9a05f78 Nov 01 06:14:56.217417 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:56.217401 4385 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:14:56.237691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:56.237662 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwpvc" event={"ID":"045ea95f-c9de-4c1e-af56-ff3712203d6a","Type":"ContainerStarted","Data":"9dbaeb7ccbb80ff9325a812dcd77bf7aaa3b59874d4009586514ab96e9a05f78"} Nov 01 06:14:56.324392 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:56.324357 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:56.472287 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:56.472187 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:56.578996 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:56.578968 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:57.207117 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:57.207082 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-xjhwv"] Nov 01 06:14:57.214293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:57.214275 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-xjhwv" Nov 01 06:14:57.243283 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:57.243251 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:57.244171 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:57.244154 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd" Nov 01 06:14:57.262801 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:57.262777 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-xjhwv"] Nov 01 06:14:57.266277 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:57.266250 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njvj4\" (UniqueName: \"kubernetes.io/projected/e624ef7d-aae2-434d-9b34-814e4bdff132-kube-api-access-njvj4\") pod \"authorino-57f7f8659f-xjhwv\" (UID: \"e624ef7d-aae2-434d-9b34-814e4bdff132\") " pod="kuadrant-system/authorino-57f7f8659f-xjhwv" Nov 01 06:14:57.266709 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:57.266688 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-f6k9r\"" Nov 01 06:14:57.367284 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:57.367252 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njvj4\" (UniqueName: \"kubernetes.io/projected/e624ef7d-aae2-434d-9b34-814e4bdff132-kube-api-access-njvj4\") pod \"authorino-57f7f8659f-xjhwv\" (UID: \"e624ef7d-aae2-434d-9b34-814e4bdff132\") " pod="kuadrant-system/authorino-57f7f8659f-xjhwv" Nov 01 06:14:57.379658 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:57.379630 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:57.397273 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:57.397237 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ggd22"] Nov 01 06:14:57.403184 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:57.403155 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ggd22" Nov 01 06:14:57.425658 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:57.425630 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njvj4\" (UniqueName: \"kubernetes.io/projected/e624ef7d-aae2-434d-9b34-814e4bdff132-kube-api-access-njvj4\") pod \"authorino-57f7f8659f-xjhwv\" (UID: \"e624ef7d-aae2-434d-9b34-814e4bdff132\") " pod="kuadrant-system/authorino-57f7f8659f-xjhwv" Nov 01 06:14:57.456369 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:57.456336 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ggd22"] Nov 01 06:14:57.467888 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:57.467793 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-452jr\" (UniqueName: \"kubernetes.io/projected/22deef10-05d5-4567-9fcc-8e0dca75c319-kube-api-access-452jr\") pod \"authorino-56c6c9ddd5-ggd22\" (UID: \"22deef10-05d5-4567-9fcc-8e0dca75c319\") " pod="kuadrant-system/authorino-56c6c9ddd5-ggd22" Nov 01 06:14:57.527535 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:57.527503 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-xjhwv" Nov 01 06:14:57.568340 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:57.568317 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-452jr\" (UniqueName: \"kubernetes.io/projected/22deef10-05d5-4567-9fcc-8e0dca75c319-kube-api-access-452jr\") pod \"authorino-56c6c9ddd5-ggd22\" (UID: \"22deef10-05d5-4567-9fcc-8e0dca75c319\") " pod="kuadrant-system/authorino-56c6c9ddd5-ggd22" Nov 01 06:14:57.575675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:57.575648 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:57.624939 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:57.624909 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-452jr\" (UniqueName: \"kubernetes.io/projected/22deef10-05d5-4567-9fcc-8e0dca75c319-kube-api-access-452jr\") pod \"authorino-56c6c9ddd5-ggd22\" (UID: \"22deef10-05d5-4567-9fcc-8e0dca75c319\") " pod="kuadrant-system/authorino-56c6c9ddd5-ggd22" Nov 01 06:14:57.717584 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:57.717550 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ggd22" Nov 01 06:14:57.923340 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:57.923315 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-xjhwv"] Nov 01 06:14:57.924640 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:14:57.924613 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode624ef7d_aae2_434d_9b34_814e4bdff132.slice/crio-8aeaadb66b14a2ecd0cc98d0e09d500c272b69a53bb22a7128551203235a2b5e WatchSource:0}: Error finding container 8aeaadb66b14a2ecd0cc98d0e09d500c272b69a53bb22a7128551203235a2b5e: Status 404 returned error can't find the container with id 8aeaadb66b14a2ecd0cc98d0e09d500c272b69a53bb22a7128551203235a2b5e Nov 01 06:14:58.099530 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:58.099501 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ggd22"] Nov 01 06:14:58.101529 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:14:58.101485 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22deef10_05d5_4567_9fcc_8e0dca75c319.slice/crio-3eec352c269bb208139d0df01052ae8550c027f110ca12666b7b06aa68145236 WatchSource:0}: Error finding container 3eec352c269bb208139d0df01052ae8550c027f110ca12666b7b06aa68145236: Status 404 returned error can't find the container with id 3eec352c269bb208139d0df01052ae8550c027f110ca12666b7b06aa68145236 Nov 01 06:14:58.248458 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:58.248392 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:58.256589 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:58.256552 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ggd22" event={"ID":"22deef10-05d5-4567-9fcc-8e0dca75c319","Type":"ContainerStarted","Data":"3eec352c269bb208139d0df01052ae8550c027f110ca12666b7b06aa68145236"} Nov 01 06:14:58.260792 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:58.260726 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-xjhwv" event={"ID":"e624ef7d-aae2-434d-9b34-814e4bdff132","Type":"ContainerStarted","Data":"8aeaadb66b14a2ecd0cc98d0e09d500c272b69a53bb22a7128551203235a2b5e"} Nov 01 06:14:58.393631 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:58.393517 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:58.847472 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:58.847438 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:59.003021 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:59.002985 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:59.203308 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:59.203234 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg" Nov 01 06:14:59.452524 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:59.452495 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:59.529770 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:59.529692 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj"] Nov 01 06:14:59.535576 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:59.535551 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj" Nov 01 06:14:59.585806 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:59.585771 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjcmb\" (UniqueName: \"kubernetes.io/projected/2eb5e419-7160-44b3-b69a-5f9fc85b6609-kube-api-access-kjcmb\") pod \"kuadrant-operator-controller-manager-678d87766f-jpshj\" (UID: \"2eb5e419-7160-44b3-b69a-5f9fc85b6609\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj" Nov 01 06:14:59.586017 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:59.585963 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2eb5e419-7160-44b3-b69a-5f9fc85b6609-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jpshj\" (UID: \"2eb5e419-7160-44b3-b69a-5f9fc85b6609\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj" Nov 01 06:14:59.615933 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:59.615889 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:14:59.644222 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:59.644190 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj"] Nov 01 06:14:59.687174 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:59.687143 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2eb5e419-7160-44b3-b69a-5f9fc85b6609-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jpshj\" (UID: \"2eb5e419-7160-44b3-b69a-5f9fc85b6609\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj" Nov 01 06:14:59.687360 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:59.687207 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjcmb\" (UniqueName: \"kubernetes.io/projected/2eb5e419-7160-44b3-b69a-5f9fc85b6609-kube-api-access-kjcmb\") pod \"kuadrant-operator-controller-manager-678d87766f-jpshj\" (UID: \"2eb5e419-7160-44b3-b69a-5f9fc85b6609\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj" Nov 01 06:14:59.687786 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:59.687755 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2eb5e419-7160-44b3-b69a-5f9fc85b6609-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jpshj\" (UID: \"2eb5e419-7160-44b3-b69a-5f9fc85b6609\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj" Nov 01 06:14:59.748523 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:59.748490 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjcmb\" (UniqueName: \"kubernetes.io/projected/2eb5e419-7160-44b3-b69a-5f9fc85b6609-kube-api-access-kjcmb\") pod \"kuadrant-operator-controller-manager-678d87766f-jpshj\" (UID: \"2eb5e419-7160-44b3-b69a-5f9fc85b6609\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj" Nov 01 06:14:59.852608 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:14:59.852575 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj" Nov 01 06:15:00.345398 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:00.344956 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj"] Nov 01 06:15:00.346400 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:15:00.346376 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb5e419_7160_44b3_b69a_5f9fc85b6609.slice/crio-ae7fdc88b3bf81d29b9f73c3cf93322c22a3ed07bdf3483e8e6e07858e2c396f WatchSource:0}: Error finding container ae7fdc88b3bf81d29b9f73c3cf93322c22a3ed07bdf3483e8e6e07858e2c396f: Status 404 returned error can't find the container with id ae7fdc88b3bf81d29b9f73c3cf93322c22a3ed07bdf3483e8e6e07858e2c396f Nov 01 06:15:01.155236 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:01.155201 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:01.213313 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:01.213278 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6jbbb" Nov 01 06:15:01.283219 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:01.283087 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ggd22" event={"ID":"22deef10-05d5-4567-9fcc-8e0dca75c319","Type":"ContainerStarted","Data":"3ee451af93c83cbf2221efe4c0ac12baf6017be5c375086d0bd6e650adbd13d4"} Nov 01 06:15:01.285070 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:01.285023 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-xjhwv" event={"ID":"e624ef7d-aae2-434d-9b34-814e4bdff132","Type":"ContainerStarted","Data":"4a9e48b097fbe317a9fffb9408adcf97a30c35518fec300fee13dfbbc1db9b53"} Nov 01 06:15:01.287006 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:01.286981 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj" event={"ID":"2eb5e419-7160-44b3-b69a-5f9fc85b6609","Type":"ContainerStarted","Data":"077fb3c71897f2cbc358b6a73eecda1861c1a14abe5589cea111099a8c4cd792"} Nov 01 06:15:01.287120 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:01.287024 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj" event={"ID":"2eb5e419-7160-44b3-b69a-5f9fc85b6609","Type":"ContainerStarted","Data":"ae7fdc88b3bf81d29b9f73c3cf93322c22a3ed07bdf3483e8e6e07858e2c396f"} Nov 01 06:15:01.287120 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:01.287078 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj" Nov 01 06:15:01.368252 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:01.368215 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:01.442509 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:01.442458 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-ggd22" podStartSLOduration=1.513247564 podStartE2EDuration="4.442443673s" totalImagesPullingTime="2.929196109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:01.440721332 +0000 UTC m=+311.204521190" watchObservedRunningTime="2026-11-01 06:15:01.442443673 +0000 UTC m=+311.206243529" Nov 01 06:15:01.467004 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:01.466958 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-xjhwv"] Nov 01 06:15:01.566619 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:01.566498 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj" podStartSLOduration=2.566475965 podStartE2EDuration="2.566475965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:01.564347373 +0000 UTC m=+311.328147237" watchObservedRunningTime="2026-11-01 06:15:01.566475965 +0000 UTC m=+311.330275831" Nov 01 06:15:01.686786 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:01.686726 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-xjhwv" podStartSLOduration=1.5865753009999999 podStartE2EDuration="4.686704833s" totalImagesPullingTime="3.100129532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:01.686637016 +0000 UTC m=+311.450436874" watchObservedRunningTime="2026-11-01 06:15:01.686704833 +0000 UTC m=+311.450504696" Nov 01 06:15:03.300329 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:03.300294 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwpvc" event={"ID":"045ea95f-c9de-4c1e-af56-ff3712203d6a","Type":"ContainerStarted","Data":"fcb8afd0c8964bdad979018aff00ac267890113fd61d2b4aad1239628a54c290"} Nov 01 06:15:03.300761 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:03.300410 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-xjhwv" podUID="e624ef7d-aae2-434d-9b34-814e4bdff132" containerName="authorino" containerID="cri-o://4a9e48b097fbe317a9fffb9408adcf97a30c35518fec300fee13dfbbc1db9b53" gracePeriod=30 Nov 01 06:15:03.530542 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:03.530492 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwpvc" podStartSLOduration=2.311899019 podStartE2EDuration="8.530476856s" totalImagesPullingTime="6.218577837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:03.526956132 +0000 UTC m=+313.290755990" watchObservedRunningTime="2026-11-01 06:15:03.530476856 +0000 UTC m=+313.294276712" Nov 01 06:15:03.556744 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:03.556667 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-xjhwv" Nov 01 06:15:03.631177 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:03.631138 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e624ef7d-aae2-434d-9b34-814e4bdff132-kube-api-access-njvj4\" (UniqueName: \"kubernetes.io/projected/e624ef7d-aae2-434d-9b34-814e4bdff132-kube-api-access-njvj4\") pod \"e624ef7d-aae2-434d-9b34-814e4bdff132\" (UID: \"e624ef7d-aae2-434d-9b34-814e4bdff132\") " Nov 01 06:15:03.633519 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:03.633495 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e624ef7d-aae2-434d-9b34-814e4bdff132-kube-api-access-njvj4" pod "e624ef7d-aae2-434d-9b34-814e4bdff132" (UID: "e624ef7d-aae2-434d-9b34-814e4bdff132"). InnerVolumeSpecName "kube-api-access-njvj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:15:03.732278 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:03.732236 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njvj4\" (UniqueName: \"kubernetes.io/projected/e624ef7d-aae2-434d-9b34-814e4bdff132-kube-api-access-njvj4\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:04.306402 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:04.306362 4385 generic.go:356] "Generic (PLEG): container finished" podID="e624ef7d-aae2-434d-9b34-814e4bdff132" containerID="4a9e48b097fbe317a9fffb9408adcf97a30c35518fec300fee13dfbbc1db9b53" exitCode=0 Nov 01 06:15:04.306813 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:04.306413 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-xjhwv" Nov 01 06:15:04.306813 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:04.306453 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-xjhwv" event={"ID":"e624ef7d-aae2-434d-9b34-814e4bdff132","Type":"ContainerDied","Data":"4a9e48b097fbe317a9fffb9408adcf97a30c35518fec300fee13dfbbc1db9b53"} Nov 01 06:15:04.306813 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:04.306515 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-xjhwv" event={"ID":"e624ef7d-aae2-434d-9b34-814e4bdff132","Type":"ContainerDied","Data":"8aeaadb66b14a2ecd0cc98d0e09d500c272b69a53bb22a7128551203235a2b5e"} Nov 01 06:15:04.306813 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:04.306556 4385 scope.go:122] "RemoveContainer" containerID="4a9e48b097fbe317a9fffb9408adcf97a30c35518fec300fee13dfbbc1db9b53" Nov 01 06:15:04.316322 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:04.316304 4385 scope.go:122] "RemoveContainer" containerID="4a9e48b097fbe317a9fffb9408adcf97a30c35518fec300fee13dfbbc1db9b53" Nov 01 06:15:04.316629 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:15:04.316608 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a9e48b097fbe317a9fffb9408adcf97a30c35518fec300fee13dfbbc1db9b53\": container with ID starting with 4a9e48b097fbe317a9fffb9408adcf97a30c35518fec300fee13dfbbc1db9b53 not found: ID does not exist" containerID="4a9e48b097fbe317a9fffb9408adcf97a30c35518fec300fee13dfbbc1db9b53" Nov 01 06:15:04.316675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:04.316639 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a9e48b097fbe317a9fffb9408adcf97a30c35518fec300fee13dfbbc1db9b53"} err="failed to get container status \"4a9e48b097fbe317a9fffb9408adcf97a30c35518fec300fee13dfbbc1db9b53\": rpc error: code = NotFound desc = could not find container \"4a9e48b097fbe317a9fffb9408adcf97a30c35518fec300fee13dfbbc1db9b53\": container with ID starting with 4a9e48b097fbe317a9fffb9408adcf97a30c35518fec300fee13dfbbc1db9b53 not found: ID does not exist" Nov 01 06:15:04.468357 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:04.468320 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-xjhwv"] Nov 01 06:15:04.470441 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:04.470424 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-xjhwv"] Nov 01 06:15:06.382261 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:06.382225 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e624ef7d-aae2-434d-9b34-814e4bdff132" path="/var/lib/kubelet/pods/e624ef7d-aae2-434d-9b34-814e4bdff132/volumes" Nov 01 06:15:06.778682 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:15:06.778597 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:15:07.267915 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:07.267881 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:07.470784 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:07.470752 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:09.840408 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:09.840371 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:09.990146 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:09.990112 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:12.263416 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:12.263378 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:12.296206 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:12.296177 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj" Nov 01 06:15:12.405176 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:12.405142 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:12.518739 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:12.518662 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg"] Nov 01 06:15:12.518954 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:12.518928 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg" podUID="7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f" containerName="manager" containerID="cri-o://0fc02ebb47f39da93ee79d1d57c52bc08ecb48120b5aa1b0142dcda2d43d0fa9" gracePeriod=10 Nov 01 06:15:14.073987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:14.073963 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg" Nov 01 06:15:14.125608 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:14.125535 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f-extensions-socket-volume\") pod \"7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f\" (UID: \"7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f\") " Nov 01 06:15:14.125608 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:14.125576 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f-kube-api-access-pnxsp\" (UniqueName: \"kubernetes.io/projected/7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f-kube-api-access-pnxsp\") pod \"7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f\" (UID: \"7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f\") " Nov 01 06:15:14.125983 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:14.125958 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f-extensions-socket-volume" pod "7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f" (UID: "7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:15:14.127744 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:14.127722 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f-kube-api-access-pnxsp" pod "7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f" (UID: "7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f"). InnerVolumeSpecName "kube-api-access-pnxsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:15:14.226407 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:14.226381 4385 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f-extensions-socket-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:14.226407 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:14.226409 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnxsp\" (UniqueName: \"kubernetes.io/projected/7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f-kube-api-access-pnxsp\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:15:14.363735 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:14.363698 4385 generic.go:356] "Generic (PLEG): container finished" podID="7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f" containerID="0fc02ebb47f39da93ee79d1d57c52bc08ecb48120b5aa1b0142dcda2d43d0fa9" exitCode=0 Nov 01 06:15:14.363902 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:14.363753 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg" Nov 01 06:15:14.363902 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:14.363782 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg" event={"ID":"7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f","Type":"ContainerDied","Data":"0fc02ebb47f39da93ee79d1d57c52bc08ecb48120b5aa1b0142dcda2d43d0fa9"} Nov 01 06:15:14.363902 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:14.363856 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg" event={"ID":"7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f","Type":"ContainerDied","Data":"acb7e9458c15108c4d8f12f5cbfd6a7ef0e8b82a024b95bc57bb9f72137d69d8"} Nov 01 06:15:14.363902 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:14.363881 4385 scope.go:122] "RemoveContainer" containerID="0fc02ebb47f39da93ee79d1d57c52bc08ecb48120b5aa1b0142dcda2d43d0fa9" Nov 01 06:15:14.372749 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:14.372727 4385 scope.go:122] "RemoveContainer" containerID="0fc02ebb47f39da93ee79d1d57c52bc08ecb48120b5aa1b0142dcda2d43d0fa9" Nov 01 06:15:14.373104 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:15:14.373087 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc02ebb47f39da93ee79d1d57c52bc08ecb48120b5aa1b0142dcda2d43d0fa9\": container with ID starting with 0fc02ebb47f39da93ee79d1d57c52bc08ecb48120b5aa1b0142dcda2d43d0fa9 not found: ID does not exist" containerID="0fc02ebb47f39da93ee79d1d57c52bc08ecb48120b5aa1b0142dcda2d43d0fa9" Nov 01 06:15:14.373156 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:14.373114 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc02ebb47f39da93ee79d1d57c52bc08ecb48120b5aa1b0142dcda2d43d0fa9"} err="failed to get container status \"0fc02ebb47f39da93ee79d1d57c52bc08ecb48120b5aa1b0142dcda2d43d0fa9\": rpc error: code = NotFound desc = could not find container \"0fc02ebb47f39da93ee79d1d57c52bc08ecb48120b5aa1b0142dcda2d43d0fa9\": container with ID starting with 0fc02ebb47f39da93ee79d1d57c52bc08ecb48120b5aa1b0142dcda2d43d0fa9 not found: ID does not exist" Nov 01 06:15:14.528008 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:14.527925 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg"] Nov 01 06:15:14.530181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:14.530163 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-99hfg"] Nov 01 06:15:16.382579 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:16.382544 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f" path="/var/lib/kubelet/pods/7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f/volumes" Nov 01 06:15:17.366213 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:17.366184 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:17.548614 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:17.548578 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:26.185328 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:26.185281 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:26.392172 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:26.392117 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:27.192089 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:27.192061 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:27.383400 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:27.383369 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:28.785518 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:28.785467 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:28.945248 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:28.945205 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:31.795309 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:31.795264 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:31.945154 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:31.945102 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:35.916454 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:35.916421 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:36.778208 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:15:36.778170 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:15:37.384091 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:37.384055 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:37.521619 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:37.521579 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:38.180715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:38.180684 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:38.332914 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:38.332872 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:38.491367 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:38.491280 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:39.470337 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:39.470301 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:39.634741 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:39.634708 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:40.095908 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.095875 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-bgkgt"] Nov 01 06:15:40.096232 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.096219 4385 cpu_manager.go:414] "containerMap: removing container" podUID="e624ef7d-aae2-434d-9b34-814e4bdff132" containerName="authorino" Nov 01 06:15:40.096281 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.096233 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e624ef7d-aae2-434d-9b34-814e4bdff132" containerName="authorino" Nov 01 06:15:40.096281 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.096253 4385 cpu_manager.go:414] "containerMap: removing container" podUID="7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f" containerName="manager" Nov 01 06:15:40.096281 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.096258 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f" containerName="manager" Nov 01 06:15:40.096382 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.096321 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="e624ef7d-aae2-434d-9b34-814e4bdff132" containerName="authorino" Nov 01 06:15:40.096382 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.096331 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab5f745-99de-4cb5-8c8d-c0a33a3ad88f" containerName="manager" Nov 01 06:15:40.113671 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.113644 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bgkgt" Nov 01 06:15:40.158192 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.158160 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-bgkgt"] Nov 01 06:15:40.167140 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.167112 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:15:40.186760 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.186730 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:40.204953 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.204928 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:15:40.204953 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.204934 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 01 06:15:40.205109 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.204982 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-5xz2t\"" Nov 01 06:15:40.256136 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.256106 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/16f2382e-1541-4146-882a-44a663facc98-data\") pod \"postgres-7cdbf6dc44-bgkgt\" (UID: \"16f2382e-1541-4146-882a-44a663facc98\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bgkgt" Nov 01 06:15:40.256317 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.256158 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5gh6\" (UniqueName: \"kubernetes.io/projected/16f2382e-1541-4146-882a-44a663facc98-kube-api-access-z5gh6\") pod \"postgres-7cdbf6dc44-bgkgt\" (UID: \"16f2382e-1541-4146-882a-44a663facc98\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bgkgt" Nov 01 06:15:40.356823 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.356744 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/16f2382e-1541-4146-882a-44a663facc98-data\") pod \"postgres-7cdbf6dc44-bgkgt\" (UID: \"16f2382e-1541-4146-882a-44a663facc98\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bgkgt" Nov 01 06:15:40.356823 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.356785 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5gh6\" (UniqueName: \"kubernetes.io/projected/16f2382e-1541-4146-882a-44a663facc98-kube-api-access-z5gh6\") pod \"postgres-7cdbf6dc44-bgkgt\" (UID: \"16f2382e-1541-4146-882a-44a663facc98\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bgkgt" Nov 01 06:15:40.357154 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.357134 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/16f2382e-1541-4146-882a-44a663facc98-data\") pod \"postgres-7cdbf6dc44-bgkgt\" (UID: \"16f2382e-1541-4146-882a-44a663facc98\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bgkgt" Nov 01 06:15:40.413572 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.413538 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5gh6\" (UniqueName: \"kubernetes.io/projected/16f2382e-1541-4146-882a-44a663facc98-kube-api-access-z5gh6\") pod \"postgres-7cdbf6dc44-bgkgt\" (UID: \"16f2382e-1541-4146-882a-44a663facc98\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bgkgt" Nov 01 06:15:40.428264 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.428237 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bgkgt" Nov 01 06:15:40.795911 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:40.795890 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-bgkgt"] Nov 01 06:15:40.798111 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:15:40.798085 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f2382e_1541_4146_882a_44a663facc98.slice/crio-d3d3a4e2413c92756b0bba336ae8f4881e8184ff03b0092cb1cb3aa0f8a2cbc1 WatchSource:0}: Error finding container d3d3a4e2413c92756b0bba336ae8f4881e8184ff03b0092cb1cb3aa0f8a2cbc1: Status 404 returned error can't find the container with id d3d3a4e2413c92756b0bba336ae8f4881e8184ff03b0092cb1cb3aa0f8a2cbc1 Nov 01 06:15:41.508419 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:41.508384 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bgkgt" event={"ID":"16f2382e-1541-4146-882a-44a663facc98","Type":"ContainerStarted","Data":"d3d3a4e2413c92756b0bba336ae8f4881e8184ff03b0092cb1cb3aa0f8a2cbc1"} Nov 01 06:15:45.354533 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:45.354483 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:45.508886 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:45.508825 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:47.546474 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:47.546436 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bgkgt" event={"ID":"16f2382e-1541-4146-882a-44a663facc98","Type":"ContainerStarted","Data":"9c14b70e3dbd4f4324e37215dc0b193a0f238067b70c6e1e2bdfc97cfeefea04"} Nov 01 06:15:47.547127 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:47.546569 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bgkgt" Nov 01 06:15:47.662044 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:47.661992 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bgkgt" podStartSLOduration=1.613192011 podStartE2EDuration="7.661976744s" totalImagesPullingTime="6.048784733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:47.659929821 +0000 UTC m=+357.423729708" watchObservedRunningTime="2026-11-01 06:15:47.661976744 +0000 UTC m=+357.425776602" Nov 01 06:15:53.173162 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:53.173121 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:53.324031 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:53.323993 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:15:53.581229 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:15:53.581204 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bgkgt" Nov 01 06:16:00.436500 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.436465 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:00.491579 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.491544 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9fc89bb6-ccp7t"] Nov 01 06:16:00.496420 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.496400 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9fc89bb6-ccp7t" Nov 01 06:16:00.552461 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.552424 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 06:16:00.555430 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.555404 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9fc89bb6-ccp7t"] Nov 01 06:16:00.568172 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.568147 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 01 06:16:00.568349 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.568152 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 06:16:00.568349 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.568187 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 01 06:16:00.601720 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.601688 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 01 06:16:00.601946 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.601931 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j8r64\"" Nov 01 06:16:00.627469 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.627438 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:00.633297 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.633274 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ngg6\" (UniqueName: \"kubernetes.io/projected/0c9e3e3a-c76a-47ec-9e23-564f620db232-kube-api-access-6ngg6\") pod \"maas-controller-9fc89bb6-ccp7t\" (UID: \"0c9e3e3a-c76a-47ec-9e23-564f620db232\") " pod="opendatahub/maas-controller-9fc89bb6-ccp7t" Nov 01 06:16:00.633431 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.633322 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c9e3e3a-c76a-47ec-9e23-564f620db232-webhook-cert\") pod \"maas-controller-9fc89bb6-ccp7t\" (UID: \"0c9e3e3a-c76a-47ec-9e23-564f620db232\") " pod="opendatahub/maas-controller-9fc89bb6-ccp7t" Nov 01 06:16:00.633431 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.633416 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9e3e3a-c76a-47ec-9e23-564f620db232-metrics-certs\") pod \"maas-controller-9fc89bb6-ccp7t\" (UID: \"0c9e3e3a-c76a-47ec-9e23-564f620db232\") " pod="opendatahub/maas-controller-9fc89bb6-ccp7t" Nov 01 06:16:00.734060 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.733976 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ngg6\" (UniqueName: \"kubernetes.io/projected/0c9e3e3a-c76a-47ec-9e23-564f620db232-kube-api-access-6ngg6\") pod \"maas-controller-9fc89bb6-ccp7t\" (UID: \"0c9e3e3a-c76a-47ec-9e23-564f620db232\") " pod="opendatahub/maas-controller-9fc89bb6-ccp7t" Nov 01 06:16:00.734060 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.734040 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c9e3e3a-c76a-47ec-9e23-564f620db232-webhook-cert\") pod \"maas-controller-9fc89bb6-ccp7t\" (UID: \"0c9e3e3a-c76a-47ec-9e23-564f620db232\") " pod="opendatahub/maas-controller-9fc89bb6-ccp7t" Nov 01 06:16:00.734257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.734077 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9e3e3a-c76a-47ec-9e23-564f620db232-metrics-certs\") pod \"maas-controller-9fc89bb6-ccp7t\" (UID: \"0c9e3e3a-c76a-47ec-9e23-564f620db232\") " pod="opendatahub/maas-controller-9fc89bb6-ccp7t" Nov 01 06:16:00.736610 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.736576 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c9e3e3a-c76a-47ec-9e23-564f620db232-webhook-cert\") pod \"maas-controller-9fc89bb6-ccp7t\" (UID: \"0c9e3e3a-c76a-47ec-9e23-564f620db232\") " pod="opendatahub/maas-controller-9fc89bb6-ccp7t" Nov 01 06:16:00.736755 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.736719 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9e3e3a-c76a-47ec-9e23-564f620db232-metrics-certs\") pod \"maas-controller-9fc89bb6-ccp7t\" (UID: \"0c9e3e3a-c76a-47ec-9e23-564f620db232\") " pod="opendatahub/maas-controller-9fc89bb6-ccp7t" Nov 01 06:16:00.777363 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.777327 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:00.794940 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.794910 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ngg6\" (UniqueName: \"kubernetes.io/projected/0c9e3e3a-c76a-47ec-9e23-564f620db232-kube-api-access-6ngg6\") pod \"maas-controller-9fc89bb6-ccp7t\" (UID: \"0c9e3e3a-c76a-47ec-9e23-564f620db232\") " pod="opendatahub/maas-controller-9fc89bb6-ccp7t" Nov 01 06:16:00.809446 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.809418 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9fc89bb6-ccp7t" Nov 01 06:16:00.913470 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:00.913429 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:01.180329 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:01.180287 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9fc89bb6-ccp7t"] Nov 01 06:16:01.210758 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:16:01.210725 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9e3e3a_c76a_47ec_9e23_564f620db232.slice/crio-50c6b3a2a61db7218ba9e1964e38e7cfbf29dc0e87c50ba51c4744053f9db7ba WatchSource:0}: Error finding container 50c6b3a2a61db7218ba9e1964e38e7cfbf29dc0e87c50ba51c4744053f9db7ba: Status 404 returned error can't find the container with id 50c6b3a2a61db7218ba9e1964e38e7cfbf29dc0e87c50ba51c4744053f9db7ba Nov 01 06:16:01.615786 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:01.615750 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9fc89bb6-ccp7t" event={"ID":"0c9e3e3a-c76a-47ec-9e23-564f620db232","Type":"ContainerStarted","Data":"50c6b3a2a61db7218ba9e1964e38e7cfbf29dc0e87c50ba51c4744053f9db7ba"} Nov 01 06:16:06.778690 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:16:06.778643 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:16:08.378223 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:08.378191 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:08.624231 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:08.624193 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:08.655494 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:08.655407 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9fc89bb6-ccp7t" event={"ID":"0c9e3e3a-c76a-47ec-9e23-564f620db232","Type":"ContainerStarted","Data":"a05e1927e6e0cd643353c757d2aa077266e28c1e51d1b412f202619ca064be83"} Nov 01 06:16:08.655494 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:08.655468 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9fc89bb6-ccp7t" Nov 01 06:16:08.774401 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:08.774349 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-9fc89bb6-ccp7t" podStartSLOduration=2.006137837 podStartE2EDuration="8.77433492s" totalImagesPullingTime="6.768197083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:16:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:16:08.771791711 +0000 UTC m=+378.535591569" watchObservedRunningTime="2026-11-01 06:16:08.77433492 +0000 UTC m=+378.538134777" Nov 01 06:16:11.440694 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:11.440662 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:11.704720 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:11.704624 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:19.381323 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:19.381287 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9fc89bb6-ccp7t"] Nov 01 06:16:19.381790 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:19.381506 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-9fc89bb6-ccp7t" podUID="0c9e3e3a-c76a-47ec-9e23-564f620db232" containerName="manager" containerID="cri-o://a05e1927e6e0cd643353c757d2aa077266e28c1e51d1b412f202619ca064be83" gracePeriod=30 Nov 01 06:16:19.383202 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:19.383181 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9fc89bb6-ccp7t" Nov 01 06:16:19.711641 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:19.711615 4385 generic.go:356] "Generic (PLEG): container finished" podID="0c9e3e3a-c76a-47ec-9e23-564f620db232" containerID="a05e1927e6e0cd643353c757d2aa077266e28c1e51d1b412f202619ca064be83" exitCode=0 Nov 01 06:16:19.711762 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:19.711687 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9fc89bb6-ccp7t" event={"ID":"0c9e3e3a-c76a-47ec-9e23-564f620db232","Type":"ContainerDied","Data":"a05e1927e6e0cd643353c757d2aa077266e28c1e51d1b412f202619ca064be83"} Nov 01 06:16:19.711762 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:19.711753 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9fc89bb6-ccp7t" event={"ID":"0c9e3e3a-c76a-47ec-9e23-564f620db232","Type":"ContainerDied","Data":"50c6b3a2a61db7218ba9e1964e38e7cfbf29dc0e87c50ba51c4744053f9db7ba"} Nov 01 06:16:19.711882 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:19.711777 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50c6b3a2a61db7218ba9e1964e38e7cfbf29dc0e87c50ba51c4744053f9db7ba" Nov 01 06:16:19.723687 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:19.723666 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9fc89bb6-ccp7t" Nov 01 06:16:19.806141 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:19.806118 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9e3e3a-c76a-47ec-9e23-564f620db232-webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c9e3e3a-c76a-47ec-9e23-564f620db232-webhook-cert\") pod \"0c9e3e3a-c76a-47ec-9e23-564f620db232\" (UID: \"0c9e3e3a-c76a-47ec-9e23-564f620db232\") " Nov 01 06:16:19.806287 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:19.806157 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9e3e3a-c76a-47ec-9e23-564f620db232-kube-api-access-6ngg6\" (UniqueName: \"kubernetes.io/projected/0c9e3e3a-c76a-47ec-9e23-564f620db232-kube-api-access-6ngg6\") pod \"0c9e3e3a-c76a-47ec-9e23-564f620db232\" (UID: \"0c9e3e3a-c76a-47ec-9e23-564f620db232\") " Nov 01 06:16:19.806330 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:19.806303 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9e3e3a-c76a-47ec-9e23-564f620db232-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9e3e3a-c76a-47ec-9e23-564f620db232-metrics-certs\") pod \"0c9e3e3a-c76a-47ec-9e23-564f620db232\" (UID: \"0c9e3e3a-c76a-47ec-9e23-564f620db232\") " Nov 01 06:16:19.808431 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:19.808395 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9e3e3a-c76a-47ec-9e23-564f620db232-metrics-certs" pod "0c9e3e3a-c76a-47ec-9e23-564f620db232" (UID: "0c9e3e3a-c76a-47ec-9e23-564f620db232"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:16:19.808538 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:19.808436 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9e3e3a-c76a-47ec-9e23-564f620db232-kube-api-access-6ngg6" pod "0c9e3e3a-c76a-47ec-9e23-564f620db232" (UID: "0c9e3e3a-c76a-47ec-9e23-564f620db232"). InnerVolumeSpecName "kube-api-access-6ngg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:16:19.808538 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:19.808459 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9e3e3a-c76a-47ec-9e23-564f620db232-webhook-cert" pod "0c9e3e3a-c76a-47ec-9e23-564f620db232" (UID: "0c9e3e3a-c76a-47ec-9e23-564f620db232"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:16:19.907496 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:19.907414 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ngg6\" (UniqueName: \"kubernetes.io/projected/0c9e3e3a-c76a-47ec-9e23-564f620db232-kube-api-access-6ngg6\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:19.907496 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:19.907446 4385 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c9e3e3a-c76a-47ec-9e23-564f620db232-metrics-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:19.907496 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:19.907456 4385 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c9e3e3a-c76a-47ec-9e23-564f620db232-webhook-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:20.716155 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:20.716080 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9fc89bb6-ccp7t" Nov 01 06:16:20.878185 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:20.878148 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9fc89bb6-ccp7t"] Nov 01 06:16:20.927750 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:20.927721 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-9fc89bb6-ccp7t"] Nov 01 06:16:22.382446 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:22.382412 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c9e3e3a-c76a-47ec-9e23-564f620db232" path="/var/lib/kubelet/pods/0c9e3e3a-c76a-47ec-9e23-564f620db232/volumes" Nov 01 06:16:24.418618 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:24.418587 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:24.591218 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:24.591184 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:24.758170 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:24.758080 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:24.913408 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:24.913370 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:25.055717 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.055618 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:25.208875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.208811 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:25.252079 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.252035 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-k9s7l"] Nov 01 06:16:25.252715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.252692 4385 cpu_manager.go:414] "containerMap: removing container" podUID="0c9e3e3a-c76a-47ec-9e23-564f620db232" containerName="manager" Nov 01 06:16:25.252825 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.252718 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9e3e3a-c76a-47ec-9e23-564f620db232" containerName="manager" Nov 01 06:16:25.252825 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.252809 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9e3e3a-c76a-47ec-9e23-564f620db232" containerName="manager" Nov 01 06:16:25.255843 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.255809 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-k9s7l" Nov 01 06:16:25.356855 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.356799 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34d1d6b5-e7a8-43b0-ba57-4ea70b9d1169-cert\") pod \"kserve-controller-manager-776cc66d47-k9s7l\" (UID: \"34d1d6b5-e7a8-43b0-ba57-4ea70b9d1169\") " pod="opendatahub/kserve-controller-manager-776cc66d47-k9s7l" Nov 01 06:16:25.357035 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.356933 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c562d\" (UniqueName: \"kubernetes.io/projected/34d1d6b5-e7a8-43b0-ba57-4ea70b9d1169-kube-api-access-c562d\") pod \"kserve-controller-manager-776cc66d47-k9s7l\" (UID: \"34d1d6b5-e7a8-43b0-ba57-4ea70b9d1169\") " pod="opendatahub/kserve-controller-manager-776cc66d47-k9s7l" Nov 01 06:16:25.358356 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.358331 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 06:16:25.358356 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.358355 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-cbm6h\"" Nov 01 06:16:25.358571 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.358373 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 06:16:25.358571 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.358382 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 01 06:16:25.361031 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.361008 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-k9s7l"] Nov 01 06:16:25.371006 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.370977 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:25.457667 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.457638 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34d1d6b5-e7a8-43b0-ba57-4ea70b9d1169-cert\") pod \"kserve-controller-manager-776cc66d47-k9s7l\" (UID: \"34d1d6b5-e7a8-43b0-ba57-4ea70b9d1169\") " pod="opendatahub/kserve-controller-manager-776cc66d47-k9s7l" Nov 01 06:16:25.458088 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.457692 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c562d\" (UniqueName: \"kubernetes.io/projected/34d1d6b5-e7a8-43b0-ba57-4ea70b9d1169-kube-api-access-c562d\") pod \"kserve-controller-manager-776cc66d47-k9s7l\" (UID: \"34d1d6b5-e7a8-43b0-ba57-4ea70b9d1169\") " pod="opendatahub/kserve-controller-manager-776cc66d47-k9s7l" Nov 01 06:16:25.460245 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.460224 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34d1d6b5-e7a8-43b0-ba57-4ea70b9d1169-cert\") pod \"kserve-controller-manager-776cc66d47-k9s7l\" (UID: \"34d1d6b5-e7a8-43b0-ba57-4ea70b9d1169\") " pod="opendatahub/kserve-controller-manager-776cc66d47-k9s7l" Nov 01 06:16:25.518377 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.518345 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c562d\" (UniqueName: \"kubernetes.io/projected/34d1d6b5-e7a8-43b0-ba57-4ea70b9d1169-kube-api-access-c562d\") pod \"kserve-controller-manager-776cc66d47-k9s7l\" (UID: \"34d1d6b5-e7a8-43b0-ba57-4ea70b9d1169\") " pod="opendatahub/kserve-controller-manager-776cc66d47-k9s7l" Nov 01 06:16:25.554282 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.554251 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:25.567324 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.567296 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-k9s7l" Nov 01 06:16:25.602671 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.602639 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-4nlhs"] Nov 01 06:16:25.606204 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.606177 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-4nlhs" Nov 01 06:16:25.660463 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.660432 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-4nlhs"] Nov 01 06:16:25.663314 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.663287 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 01 06:16:25.663440 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.663345 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-tft88\"" Nov 01 06:16:25.663440 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.663369 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 01 06:16:25.759748 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.759711 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b76cbc0-abbe-45e8-a061-e44e87c2f782-cert\") pod \"odh-model-controller-769875db56-4nlhs\" (UID: \"9b76cbc0-abbe-45e8-a061-e44e87c2f782\") " pod="opendatahub/odh-model-controller-769875db56-4nlhs" Nov 01 06:16:25.759956 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.759894 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48mvr\" (UniqueName: \"kubernetes.io/projected/9b76cbc0-abbe-45e8-a061-e44e87c2f782-kube-api-access-48mvr\") pod \"odh-model-controller-769875db56-4nlhs\" (UID: \"9b76cbc0-abbe-45e8-a061-e44e87c2f782\") " pod="opendatahub/odh-model-controller-769875db56-4nlhs" Nov 01 06:16:25.759956 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.759930 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b76cbc0-abbe-45e8-a061-e44e87c2f782-metrics-certs\") pod \"odh-model-controller-769875db56-4nlhs\" (UID: \"9b76cbc0-abbe-45e8-a061-e44e87c2f782\") " pod="opendatahub/odh-model-controller-769875db56-4nlhs" Nov 01 06:16:25.860531 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.860494 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b76cbc0-abbe-45e8-a061-e44e87c2f782-cert\") pod \"odh-model-controller-769875db56-4nlhs\" (UID: \"9b76cbc0-abbe-45e8-a061-e44e87c2f782\") " pod="opendatahub/odh-model-controller-769875db56-4nlhs" Nov 01 06:16:25.860688 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.860599 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48mvr\" (UniqueName: \"kubernetes.io/projected/9b76cbc0-abbe-45e8-a061-e44e87c2f782-kube-api-access-48mvr\") pod \"odh-model-controller-769875db56-4nlhs\" (UID: \"9b76cbc0-abbe-45e8-a061-e44e87c2f782\") " pod="opendatahub/odh-model-controller-769875db56-4nlhs" Nov 01 06:16:25.860688 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.860636 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b76cbc0-abbe-45e8-a061-e44e87c2f782-metrics-certs\") pod \"odh-model-controller-769875db56-4nlhs\" (UID: \"9b76cbc0-abbe-45e8-a061-e44e87c2f782\") " pod="opendatahub/odh-model-controller-769875db56-4nlhs" Nov 01 06:16:25.863320 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.863295 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b76cbc0-abbe-45e8-a061-e44e87c2f782-cert\") pod \"odh-model-controller-769875db56-4nlhs\" (UID: \"9b76cbc0-abbe-45e8-a061-e44e87c2f782\") " pod="opendatahub/odh-model-controller-769875db56-4nlhs" Nov 01 06:16:25.863320 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.863309 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b76cbc0-abbe-45e8-a061-e44e87c2f782-metrics-certs\") pod \"odh-model-controller-769875db56-4nlhs\" (UID: \"9b76cbc0-abbe-45e8-a061-e44e87c2f782\") " pod="opendatahub/odh-model-controller-769875db56-4nlhs" Nov 01 06:16:25.917532 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.917465 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48mvr\" (UniqueName: \"kubernetes.io/projected/9b76cbc0-abbe-45e8-a061-e44e87c2f782-kube-api-access-48mvr\") pod \"odh-model-controller-769875db56-4nlhs\" (UID: \"9b76cbc0-abbe-45e8-a061-e44e87c2f782\") " pod="opendatahub/odh-model-controller-769875db56-4nlhs" Nov 01 06:16:25.921282 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.921259 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-4nlhs" Nov 01 06:16:25.940507 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:25.940484 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-k9s7l"] Nov 01 06:16:25.941992 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:16:25.941969 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d1d6b5_e7a8_43b0_ba57_4ea70b9d1169.slice/crio-97c2825a73604cde598a44a7ac68242dd14a67f20bd8b29b26cd907b277af58c WatchSource:0}: Error finding container 97c2825a73604cde598a44a7ac68242dd14a67f20bd8b29b26cd907b277af58c: Status 404 returned error can't find the container with id 97c2825a73604cde598a44a7ac68242dd14a67f20bd8b29b26cd907b277af58c Nov 01 06:16:26.288609 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:26.288582 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-4nlhs"] Nov 01 06:16:26.289654 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:16:26.289620 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b76cbc0_abbe_45e8_a061_e44e87c2f782.slice/crio-4e223a0c834ba42b985fd612f8ef66ec39e7a10349e02bca0c50629dec76fd1d WatchSource:0}: Error finding container 4e223a0c834ba42b985fd612f8ef66ec39e7a10349e02bca0c50629dec76fd1d: Status 404 returned error can't find the container with id 4e223a0c834ba42b985fd612f8ef66ec39e7a10349e02bca0c50629dec76fd1d Nov 01 06:16:26.638374 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:26.638325 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:26.747814 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:26.747782 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-4nlhs" event={"ID":"9b76cbc0-abbe-45e8-a061-e44e87c2f782","Type":"ContainerStarted","Data":"4e223a0c834ba42b985fd612f8ef66ec39e7a10349e02bca0c50629dec76fd1d"} Nov 01 06:16:26.749050 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:26.749024 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-k9s7l" event={"ID":"34d1d6b5-e7a8-43b0-ba57-4ea70b9d1169","Type":"ContainerStarted","Data":"97c2825a73604cde598a44a7ac68242dd14a67f20bd8b29b26cd907b277af58c"} Nov 01 06:16:29.770912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:29.770869 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-4nlhs" event={"ID":"9b76cbc0-abbe-45e8-a061-e44e87c2f782","Type":"ContainerStarted","Data":"3d1ab26a31192ef2268993d475803e8785923246830ba7209b3794000b32fff2"} Nov 01 06:16:29.771374 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:29.770997 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-4nlhs" Nov 01 06:16:29.772619 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:29.772592 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-k9s7l" event={"ID":"34d1d6b5-e7a8-43b0-ba57-4ea70b9d1169","Type":"ContainerStarted","Data":"0365b501482b40cbabbd83121767d96953e7ba3cc5e92558b477a4b9a0ca576f"} Nov 01 06:16:29.772762 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:29.772640 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-k9s7l" Nov 01 06:16:29.885048 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:29.884999 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-4nlhs" podStartSLOduration=1.607711572 podStartE2EDuration="4.884986028s" totalImagesPullingTime="3.277274456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:16:29.882558914 +0000 UTC m=+399.646358771" watchObservedRunningTime="2026-11-01 06:16:29.884986028 +0000 UTC m=+399.648785884" Nov 01 06:16:29.995550 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:29.995336 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-k9s7l" podStartSLOduration=1.672307515 podStartE2EDuration="4.995317807s" totalImagesPullingTime="3.323010292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:16:29.994752357 +0000 UTC m=+399.758552214" watchObservedRunningTime="2026-11-01 06:16:29.995317807 +0000 UTC m=+399.759117664" Nov 01 06:16:31.780756 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:31.780660 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:31.926869 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:31.926821 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:32.071442 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:32.071363 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:32.211076 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:32.211044 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:32.352698 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:32.352664 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:32.497518 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:32.497485 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:32.698396 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:32.698304 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:32.848628 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:32.848592 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:32.991216 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:32.991128 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:36.238941 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:36.238899 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:36.779128 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:16:36.779093 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:16:37.973751 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:37.973712 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:38.211725 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:38.211686 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:40.010645 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:40.010598 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:40.172707 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:40.172674 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:40.317715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:40.317628 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:40.460863 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:40.460798 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:40.779969 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:40.779943 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-4nlhs" Nov 01 06:16:53.900905 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:53.900871 4385 generic.go:356] "Generic (PLEG): container finished" podID="bf4f43cd-9da8-4bd6-9b49-9c7329b5480a" containerID="2c03f0b7f2c143b073fff854d013af9f7a5d652c62311a91e8b65a8ebfc7f3fd" exitCode=0 Nov 01 06:16:53.901335 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:53.900949 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" event={"ID":"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a","Type":"ContainerDied","Data":"2c03f0b7f2c143b073fff854d013af9f7a5d652c62311a91e8b65a8ebfc7f3fd"} Nov 01 06:16:53.901377 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:53.901364 4385 scope.go:122] "RemoveContainer" containerID="2c03f0b7f2c143b073fff854d013af9f7a5d652c62311a91e8b65a8ebfc7f3fd" Nov 01 06:16:53.902460 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:53.902433 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:16:53.902556 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:53.902497 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:16:53.902556 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:53.902541 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:16:54.837281 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:54.837247 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:54.908155 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:54.908118 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" event={"ID":"bf4f43cd-9da8-4bd6-9b49-9c7329b5480a","Type":"ContainerStarted","Data":"017b8406669576350cec0d4193af3afa8d2ceefedf7dd265ccf0dda1a9989fa3"} Nov 01 06:16:55.129888 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.129801 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7whnd"] Nov 01 06:16:55.132089 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.132073 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-7whnd" Nov 01 06:16:55.155025 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.154995 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:16:55.198640 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.198522 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7whnd"] Nov 01 06:16:55.247510 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.247470 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkxmd\" (UniqueName: \"kubernetes.io/projected/caeda150-ffc3-4c77-80f9-7fae36ea9263-kube-api-access-pkxmd\") pod \"authorino-6f6944bcdd-7whnd\" (UID: \"caeda150-ffc3-4c77-80f9-7fae36ea9263\") " pod="kuadrant-system/authorino-6f6944bcdd-7whnd" Nov 01 06:16:55.348481 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.348442 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkxmd\" (UniqueName: \"kubernetes.io/projected/caeda150-ffc3-4c77-80f9-7fae36ea9263-kube-api-access-pkxmd\") pod \"authorino-6f6944bcdd-7whnd\" (UID: \"caeda150-ffc3-4c77-80f9-7fae36ea9263\") " pod="kuadrant-system/authorino-6f6944bcdd-7whnd" Nov 01 06:16:55.405048 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.404962 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkxmd\" (UniqueName: \"kubernetes.io/projected/caeda150-ffc3-4c77-80f9-7fae36ea9263-kube-api-access-pkxmd\") pod \"authorino-6f6944bcdd-7whnd\" (UID: \"caeda150-ffc3-4c77-80f9-7fae36ea9263\") " pod="kuadrant-system/authorino-6f6944bcdd-7whnd" Nov 01 06:16:55.443346 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.443315 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-7whnd" Nov 01 06:16:55.496052 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.496011 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:16:55.497999 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.497965 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8ckbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 01 06:16:55.498142 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.498040 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" podUID="bf4f43cd-9da8-4bd6-9b49-9c7329b5480a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 01 06:16:55.499673 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.499646 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7whnd"] Nov 01 06:16:55.511111 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.511075 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-779d94779c-7l5gf"] Nov 01 06:16:55.517712 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.517668 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-779d94779c-7l5gf" Nov 01 06:16:55.569514 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.569486 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-779d94779c-7l5gf"] Nov 01 06:16:55.650710 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.650676 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwgbg\" (UniqueName: \"kubernetes.io/projected/1c6a47db-4dcc-4314-8926-a898ff1ce943-kube-api-access-nwgbg\") pod \"authorino-779d94779c-7l5gf\" (UID: \"1c6a47db-4dcc-4314-8926-a898ff1ce943\") " pod="kuadrant-system/authorino-779d94779c-7l5gf" Nov 01 06:16:55.751444 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.751355 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwgbg\" (UniqueName: \"kubernetes.io/projected/1c6a47db-4dcc-4314-8926-a898ff1ce943-kube-api-access-nwgbg\") pod \"authorino-779d94779c-7l5gf\" (UID: \"1c6a47db-4dcc-4314-8926-a898ff1ce943\") " pod="kuadrant-system/authorino-779d94779c-7l5gf" Nov 01 06:16:55.807801 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.807772 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwgbg\" (UniqueName: \"kubernetes.io/projected/1c6a47db-4dcc-4314-8926-a898ff1ce943-kube-api-access-nwgbg\") pod \"authorino-779d94779c-7l5gf\" (UID: \"1c6a47db-4dcc-4314-8926-a898ff1ce943\") " pod="kuadrant-system/authorino-779d94779c-7l5gf" Nov 01 06:16:55.838910 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.838880 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-779d94779c-7l5gf" Nov 01 06:16:55.841668 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.841640 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7whnd"] Nov 01 06:16:55.843261 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:16:55.843236 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcaeda150_ffc3_4c77_80f9_7fae36ea9263.slice/crio-474a40092cdc27f814a34b9968caac0ec056fcad30fc37568a217fc0af80fd13 WatchSource:0}: Error finding container 474a40092cdc27f814a34b9968caac0ec056fcad30fc37568a217fc0af80fd13: Status 404 returned error can't find the container with id 474a40092cdc27f814a34b9968caac0ec056fcad30fc37568a217fc0af80fd13 Nov 01 06:16:55.918322 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.917639 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-7whnd" event={"ID":"caeda150-ffc3-4c77-80f9-7fae36ea9263","Type":"ContainerStarted","Data":"474a40092cdc27f814a34b9968caac0ec056fcad30fc37568a217fc0af80fd13"} Nov 01 06:16:55.953202 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.953169 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-779d94779c-7l5gf"] Nov 01 06:16:55.998076 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:55.998031 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-64c8bb8959-2zlbz"] Nov 01 06:16:56.003284 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:56.003214 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64c8bb8959-2zlbz" Nov 01 06:16:56.035722 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:56.035695 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64c8bb8959-2zlbz"] Nov 01 06:16:56.055193 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:56.055165 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 01 06:16:56.055424 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:56.055394 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgsf8\" (UniqueName: \"kubernetes.io/projected/e4749577-adda-47e0-b660-8f12cf0f08fb-kube-api-access-kgsf8\") pod \"authorino-64c8bb8959-2zlbz\" (UID: \"e4749577-adda-47e0-b660-8f12cf0f08fb\") " pod="kuadrant-system/authorino-64c8bb8959-2zlbz" Nov 01 06:16:56.055512 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:56.055456 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e4749577-adda-47e0-b660-8f12cf0f08fb-tls-cert\") pod \"authorino-64c8bb8959-2zlbz\" (UID: \"e4749577-adda-47e0-b660-8f12cf0f08fb\") " pod="kuadrant-system/authorino-64c8bb8959-2zlbz" Nov 01 06:16:56.156432 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:56.156398 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgsf8\" (UniqueName: \"kubernetes.io/projected/e4749577-adda-47e0-b660-8f12cf0f08fb-kube-api-access-kgsf8\") pod \"authorino-64c8bb8959-2zlbz\" (UID: \"e4749577-adda-47e0-b660-8f12cf0f08fb\") " pod="kuadrant-system/authorino-64c8bb8959-2zlbz" Nov 01 06:16:56.156432 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:56.156436 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e4749577-adda-47e0-b660-8f12cf0f08fb-tls-cert\") pod \"authorino-64c8bb8959-2zlbz\" (UID: \"e4749577-adda-47e0-b660-8f12cf0f08fb\") " pod="kuadrant-system/authorino-64c8bb8959-2zlbz" Nov 01 06:16:56.159321 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:56.159297 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e4749577-adda-47e0-b660-8f12cf0f08fb-tls-cert\") pod \"authorino-64c8bb8959-2zlbz\" (UID: \"e4749577-adda-47e0-b660-8f12cf0f08fb\") " pod="kuadrant-system/authorino-64c8bb8959-2zlbz" Nov 01 06:16:56.212635 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:56.212604 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgsf8\" (UniqueName: \"kubernetes.io/projected/e4749577-adda-47e0-b660-8f12cf0f08fb-kube-api-access-kgsf8\") pod \"authorino-64c8bb8959-2zlbz\" (UID: \"e4749577-adda-47e0-b660-8f12cf0f08fb\") " pod="kuadrant-system/authorino-64c8bb8959-2zlbz" Nov 01 06:16:56.237344 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:56.237317 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-779d94779c-7l5gf"] Nov 01 06:16:56.237894 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:16:56.237825 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c6a47db_4dcc_4314_8926_a898ff1ce943.slice/crio-30103ee5a4a9c82d404420c84975b62bf067bedb8be7ff0d57342e158799546a WatchSource:0}: Error finding container 30103ee5a4a9c82d404420c84975b62bf067bedb8be7ff0d57342e158799546a: Status 404 returned error can't find the container with id 30103ee5a4a9c82d404420c84975b62bf067bedb8be7ff0d57342e158799546a Nov 01 06:16:56.315418 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:56.315342 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64c8bb8959-2zlbz" Nov 01 06:16:56.495214 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:56.495177 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8ckbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 01 06:16:56.495393 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:56.495235 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" podUID="bf4f43cd-9da8-4bd6-9b49-9c7329b5480a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 01 06:16:56.700593 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:56.700562 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64c8bb8959-2zlbz"] Nov 01 06:16:56.701903 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:16:56.701873 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4749577_adda_47e0_b660_8f12cf0f08fb.slice/crio-26d3bb08156591e4ef17d123be1371ba55d379e40776c9772027db17f548c9c4 WatchSource:0}: Error finding container 26d3bb08156591e4ef17d123be1371ba55d379e40776c9772027db17f548c9c4: Status 404 returned error can't find the container with id 26d3bb08156591e4ef17d123be1371ba55d379e40776c9772027db17f548c9c4 Nov 01 06:16:56.923147 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:56.923089 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64c8bb8959-2zlbz" event={"ID":"e4749577-adda-47e0-b660-8f12cf0f08fb","Type":"ContainerStarted","Data":"26d3bb08156591e4ef17d123be1371ba55d379e40776c9772027db17f548c9c4"} Nov 01 06:16:56.924184 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:56.924161 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-779d94779c-7l5gf" event={"ID":"1c6a47db-4dcc-4314-8926-a898ff1ce943","Type":"ContainerStarted","Data":"30103ee5a4a9c82d404420c84975b62bf067bedb8be7ff0d57342e158799546a"} Nov 01 06:16:57.482722 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:57.482219 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8ckbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 01 06:16:57.482722 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:57.482280 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" podUID="bf4f43cd-9da8-4bd6-9b49-9c7329b5480a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 01 06:16:57.939341 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:57.939297 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-7whnd" event={"ID":"caeda150-ffc3-4c77-80f9-7fae36ea9263","Type":"ContainerStarted","Data":"37e663e58a7163e044a5d44abec0ed3590f70bec48d87ea3a264d5d3b7424de5"} Nov 01 06:16:57.939793 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:57.939517 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-7whnd" podUID="caeda150-ffc3-4c77-80f9-7fae36ea9263" containerName="authorino" containerID="cri-o://37e663e58a7163e044a5d44abec0ed3590f70bec48d87ea3a264d5d3b7424de5" gracePeriod=30 Nov 01 06:16:57.947380 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:57.946973 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-779d94779c-7l5gf" event={"ID":"1c6a47db-4dcc-4314-8926-a898ff1ce943","Type":"ContainerStarted","Data":"ac3e010b485e31f2d1728d831ca92a15c957f0964d5abac6cf3ff9f67b682412"} Nov 01 06:16:57.947380 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:57.947189 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-779d94779c-7l5gf" podUID="1c6a47db-4dcc-4314-8926-a898ff1ce943" containerName="authorino" containerID="cri-o://ac3e010b485e31f2d1728d831ca92a15c957f0964d5abac6cf3ff9f67b682412" gracePeriod=30 Nov 01 06:16:58.054617 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.053475 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-7whnd" podStartSLOduration=1.678674434 podStartE2EDuration="3.053459961s" totalImagesPullingTime="1.374785527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:16:58.051642483 +0000 UTC m=+427.815442339" watchObservedRunningTime="2026-11-01 06:16:58.053459961 +0000 UTC m=+427.817259817" Nov 01 06:16:58.252785 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.252764 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-7whnd" Nov 01 06:16:58.313080 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.313058 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-779d94779c-7l5gf" Nov 01 06:16:58.364975 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.364915 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-779d94779c-7l5gf" podStartSLOduration=2.065541239 podStartE2EDuration="3.36489419s" totalImagesPullingTime="1.299352951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:16:58.168036911 +0000 UTC m=+427.931836769" watchObservedRunningTime="2026-11-01 06:16:58.36489419 +0000 UTC m=+428.128694048" Nov 01 06:16:58.401803 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.400655 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caeda150-ffc3-4c77-80f9-7fae36ea9263-kube-api-access-pkxmd\" (UniqueName: \"kubernetes.io/projected/caeda150-ffc3-4c77-80f9-7fae36ea9263-kube-api-access-pkxmd\") pod \"caeda150-ffc3-4c77-80f9-7fae36ea9263\" (UID: \"caeda150-ffc3-4c77-80f9-7fae36ea9263\") " Nov 01 06:16:58.401803 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.400964 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c6a47db-4dcc-4314-8926-a898ff1ce943-kube-api-access-nwgbg\" (UniqueName: \"kubernetes.io/projected/1c6a47db-4dcc-4314-8926-a898ff1ce943-kube-api-access-nwgbg\") pod \"1c6a47db-4dcc-4314-8926-a898ff1ce943\" (UID: \"1c6a47db-4dcc-4314-8926-a898ff1ce943\") " Nov 01 06:16:58.403971 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.403907 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6a47db-4dcc-4314-8926-a898ff1ce943-kube-api-access-nwgbg" pod "1c6a47db-4dcc-4314-8926-a898ff1ce943" (UID: "1c6a47db-4dcc-4314-8926-a898ff1ce943"). InnerVolumeSpecName "kube-api-access-nwgbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:16:58.405943 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.405894 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caeda150-ffc3-4c77-80f9-7fae36ea9263-kube-api-access-pkxmd" pod "caeda150-ffc3-4c77-80f9-7fae36ea9263" (UID: "caeda150-ffc3-4c77-80f9-7fae36ea9263"). InnerVolumeSpecName "kube-api-access-pkxmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:16:58.494528 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.493931 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8ckbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 01 06:16:58.494528 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.493986 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" podUID="bf4f43cd-9da8-4bd6-9b49-9c7329b5480a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 01 06:16:58.502488 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.502454 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwgbg\" (UniqueName: \"kubernetes.io/projected/1c6a47db-4dcc-4314-8926-a898ff1ce943-kube-api-access-nwgbg\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:58.502488 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.502483 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkxmd\" (UniqueName: \"kubernetes.io/projected/caeda150-ffc3-4c77-80f9-7fae36ea9263-kube-api-access-pkxmd\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:58.504205 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.503904 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:16:58.954935 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.954900 4385 generic.go:356] "Generic (PLEG): container finished" podID="1c6a47db-4dcc-4314-8926-a898ff1ce943" containerID="ac3e010b485e31f2d1728d831ca92a15c957f0964d5abac6cf3ff9f67b682412" exitCode=0 Nov 01 06:16:58.955321 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.955042 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-779d94779c-7l5gf" event={"ID":"1c6a47db-4dcc-4314-8926-a898ff1ce943","Type":"ContainerDied","Data":"ac3e010b485e31f2d1728d831ca92a15c957f0964d5abac6cf3ff9f67b682412"} Nov 01 06:16:58.955321 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.955071 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-779d94779c-7l5gf" event={"ID":"1c6a47db-4dcc-4314-8926-a898ff1ce943","Type":"ContainerDied","Data":"30103ee5a4a9c82d404420c84975b62bf067bedb8be7ff0d57342e158799546a"} Nov 01 06:16:58.955321 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.955087 4385 scope.go:122] "RemoveContainer" containerID="ac3e010b485e31f2d1728d831ca92a15c957f0964d5abac6cf3ff9f67b682412" Nov 01 06:16:58.955321 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.955204 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-779d94779c-7l5gf" Nov 01 06:16:58.958079 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.958043 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64c8bb8959-2zlbz" event={"ID":"e4749577-adda-47e0-b660-8f12cf0f08fb","Type":"ContainerStarted","Data":"951407e4fbb8347037216777125f088470e284706ba299942e0874508b184c6e"} Nov 01 06:16:58.959552 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.959524 4385 generic.go:356] "Generic (PLEG): container finished" podID="caeda150-ffc3-4c77-80f9-7fae36ea9263" containerID="37e663e58a7163e044a5d44abec0ed3590f70bec48d87ea3a264d5d3b7424de5" exitCode=0 Nov 01 06:16:58.959659 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.959581 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-7whnd" event={"ID":"caeda150-ffc3-4c77-80f9-7fae36ea9263","Type":"ContainerDied","Data":"37e663e58a7163e044a5d44abec0ed3590f70bec48d87ea3a264d5d3b7424de5"} Nov 01 06:16:58.959659 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.959587 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-7whnd" Nov 01 06:16:58.959659 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.959607 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-7whnd" event={"ID":"caeda150-ffc3-4c77-80f9-7fae36ea9263","Type":"ContainerDied","Data":"474a40092cdc27f814a34b9968caac0ec056fcad30fc37568a217fc0af80fd13"} Nov 01 06:16:58.968022 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.967999 4385 scope.go:122] "RemoveContainer" containerID="ac3e010b485e31f2d1728d831ca92a15c957f0964d5abac6cf3ff9f67b682412" Nov 01 06:16:58.969264 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:16:58.969125 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3e010b485e31f2d1728d831ca92a15c957f0964d5abac6cf3ff9f67b682412\": container with ID starting with ac3e010b485e31f2d1728d831ca92a15c957f0964d5abac6cf3ff9f67b682412 not found: ID does not exist" containerID="ac3e010b485e31f2d1728d831ca92a15c957f0964d5abac6cf3ff9f67b682412" Nov 01 06:16:58.969264 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.969164 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3e010b485e31f2d1728d831ca92a15c957f0964d5abac6cf3ff9f67b682412"} err="failed to get container status \"ac3e010b485e31f2d1728d831ca92a15c957f0964d5abac6cf3ff9f67b682412\": rpc error: code = NotFound desc = could not find container \"ac3e010b485e31f2d1728d831ca92a15c957f0964d5abac6cf3ff9f67b682412\": container with ID starting with ac3e010b485e31f2d1728d831ca92a15c957f0964d5abac6cf3ff9f67b682412 not found: ID does not exist" Nov 01 06:16:58.969264 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:58.969188 4385 scope.go:122] "RemoveContainer" containerID="37e663e58a7163e044a5d44abec0ed3590f70bec48d87ea3a264d5d3b7424de5" Nov 01 06:16:59.008452 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.008431 4385 scope.go:122] "RemoveContainer" containerID="37e663e58a7163e044a5d44abec0ed3590f70bec48d87ea3a264d5d3b7424de5" Nov 01 06:16:59.008778 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:16:59.008758 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37e663e58a7163e044a5d44abec0ed3590f70bec48d87ea3a264d5d3b7424de5\": container with ID starting with 37e663e58a7163e044a5d44abec0ed3590f70bec48d87ea3a264d5d3b7424de5 not found: ID does not exist" containerID="37e663e58a7163e044a5d44abec0ed3590f70bec48d87ea3a264d5d3b7424de5" Nov 01 06:16:59.008967 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.008791 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37e663e58a7163e044a5d44abec0ed3590f70bec48d87ea3a264d5d3b7424de5"} err="failed to get container status \"37e663e58a7163e044a5d44abec0ed3590f70bec48d87ea3a264d5d3b7424de5\": rpc error: code = NotFound desc = could not find container \"37e663e58a7163e044a5d44abec0ed3590f70bec48d87ea3a264d5d3b7424de5\": container with ID starting with 37e663e58a7163e044a5d44abec0ed3590f70bec48d87ea3a264d5d3b7424de5 not found: ID does not exist" Nov 01 06:16:59.118513 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.118475 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-779d94779c-7l5gf"] Nov 01 06:16:59.168774 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.168728 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-779d94779c-7l5gf"] Nov 01 06:16:59.232392 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.232311 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7whnd"] Nov 01 06:16:59.236403 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.236380 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7whnd"] Nov 01 06:16:59.352120 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.352071 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-64c8bb8959-2zlbz" podStartSLOduration=3.130358645 podStartE2EDuration="4.352055645s" totalImagesPullingTime="1.221697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:16:59.351391328 +0000 UTC m=+429.115191186" watchObservedRunningTime="2026-11-01 06:16:59.352055645 +0000 UTC m=+429.115855516" Nov 01 06:16:59.400686 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.400637 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ggd22"] Nov 01 06:16:59.401490 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.401460 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-ggd22" podUID="22deef10-05d5-4567-9fcc-8e0dca75c319" containerName="authorino" containerID="cri-o://3ee451af93c83cbf2221efe4c0ac12baf6017be5c375086d0bd6e650adbd13d4" gracePeriod=30 Nov 01 06:16:59.496821 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.496290 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8ckbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Nov 01 06:16:59.496821 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.496355 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" podUID="bf4f43cd-9da8-4bd6-9b49-9c7329b5480a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Nov 01 06:16:59.652919 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.652895 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ggd22" Nov 01 06:16:59.814561 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.814471 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22deef10-05d5-4567-9fcc-8e0dca75c319-kube-api-access-452jr\" (UniqueName: \"kubernetes.io/projected/22deef10-05d5-4567-9fcc-8e0dca75c319-kube-api-access-452jr\") pod \"22deef10-05d5-4567-9fcc-8e0dca75c319\" (UID: \"22deef10-05d5-4567-9fcc-8e0dca75c319\") " Nov 01 06:16:59.816716 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.816690 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22deef10-05d5-4567-9fcc-8e0dca75c319-kube-api-access-452jr" pod "22deef10-05d5-4567-9fcc-8e0dca75c319" (UID: "22deef10-05d5-4567-9fcc-8e0dca75c319"). InnerVolumeSpecName "kube-api-access-452jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:16:59.915998 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.915964 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-452jr\" (UniqueName: \"kubernetes.io/projected/22deef10-05d5-4567-9fcc-8e0dca75c319-kube-api-access-452jr\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:59.967760 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.967724 4385 generic.go:356] "Generic (PLEG): container finished" podID="22deef10-05d5-4567-9fcc-8e0dca75c319" containerID="3ee451af93c83cbf2221efe4c0ac12baf6017be5c375086d0bd6e650adbd13d4" exitCode=0 Nov 01 06:16:59.968249 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.967784 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ggd22" Nov 01 06:16:59.968249 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.967825 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ggd22" event={"ID":"22deef10-05d5-4567-9fcc-8e0dca75c319","Type":"ContainerDied","Data":"3ee451af93c83cbf2221efe4c0ac12baf6017be5c375086d0bd6e650adbd13d4"} Nov 01 06:16:59.968249 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.967892 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ggd22" event={"ID":"22deef10-05d5-4567-9fcc-8e0dca75c319","Type":"ContainerDied","Data":"3eec352c269bb208139d0df01052ae8550c027f110ca12666b7b06aa68145236"} Nov 01 06:16:59.968249 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.967921 4385 scope.go:122] "RemoveContainer" containerID="3ee451af93c83cbf2221efe4c0ac12baf6017be5c375086d0bd6e650adbd13d4" Nov 01 06:16:59.977865 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.977820 4385 scope.go:122] "RemoveContainer" containerID="3ee451af93c83cbf2221efe4c0ac12baf6017be5c375086d0bd6e650adbd13d4" Nov 01 06:16:59.978190 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:16:59.978166 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee451af93c83cbf2221efe4c0ac12baf6017be5c375086d0bd6e650adbd13d4\": container with ID starting with 3ee451af93c83cbf2221efe4c0ac12baf6017be5c375086d0bd6e650adbd13d4 not found: ID does not exist" containerID="3ee451af93c83cbf2221efe4c0ac12baf6017be5c375086d0bd6e650adbd13d4" Nov 01 06:16:59.978282 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:16:59.978203 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee451af93c83cbf2221efe4c0ac12baf6017be5c375086d0bd6e650adbd13d4"} err="failed to get container status \"3ee451af93c83cbf2221efe4c0ac12baf6017be5c375086d0bd6e650adbd13d4\": rpc error: code = NotFound desc = could not find container \"3ee451af93c83cbf2221efe4c0ac12baf6017be5c375086d0bd6e650adbd13d4\": container with ID starting with 3ee451af93c83cbf2221efe4c0ac12baf6017be5c375086d0bd6e650adbd13d4 not found: ID does not exist" Nov 01 06:17:00.131110 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:00.131072 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ggd22"] Nov 01 06:17:00.133330 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:00.133311 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ggd22"] Nov 01 06:17:00.383515 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:00.383423 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c6a47db-4dcc-4314-8926-a898ff1ce943" path="/var/lib/kubelet/pods/1c6a47db-4dcc-4314-8926-a898ff1ce943/volumes" Nov 01 06:17:00.383876 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:00.383856 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22deef10-05d5-4567-9fcc-8e0dca75c319" path="/var/lib/kubelet/pods/22deef10-05d5-4567-9fcc-8e0dca75c319/volumes" Nov 01 06:17:00.384198 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:00.384182 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caeda150-ffc3-4c77-80f9-7fae36ea9263" path="/var/lib/kubelet/pods/caeda150-ffc3-4c77-80f9-7fae36ea9263/volumes" Nov 01 06:17:00.485220 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:00.485194 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:17:00.486246 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:00.486228 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" Nov 01 06:17:00.783274 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:00.783198 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-k9s7l" Nov 01 06:17:02.677262 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:02.677221 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:17:03.261337 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:03.261297 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:17:03.406660 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:03.406623 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:17:03.559642 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:03.559558 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:17:06.777636 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:17:06.777602 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:17:36.588972 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:36.588935 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:17:36.778741 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:17:36.778703 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:17:40.653740 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:40.653708 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-446vb"] Nov 01 06:17:40.654148 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:40.654096 4385 cpu_manager.go:414] "containerMap: removing container" podUID="caeda150-ffc3-4c77-80f9-7fae36ea9263" containerName="authorino" Nov 01 06:17:40.654148 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:40.654108 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caeda150-ffc3-4c77-80f9-7fae36ea9263" containerName="authorino" Nov 01 06:17:40.654148 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:40.654123 4385 cpu_manager.go:414] "containerMap: removing container" podUID="22deef10-05d5-4567-9fcc-8e0dca75c319" containerName="authorino" Nov 01 06:17:40.654148 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:40.654129 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22deef10-05d5-4567-9fcc-8e0dca75c319" containerName="authorino" Nov 01 06:17:40.654148 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:40.654138 4385 cpu_manager.go:414] "containerMap: removing container" podUID="1c6a47db-4dcc-4314-8926-a898ff1ce943" containerName="authorino" Nov 01 06:17:40.654148 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:40.654143 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c6a47db-4dcc-4314-8926-a898ff1ce943" containerName="authorino" Nov 01 06:17:40.654333 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:40.654195 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="caeda150-ffc3-4c77-80f9-7fae36ea9263" containerName="authorino" Nov 01 06:17:40.654333 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:40.654205 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="22deef10-05d5-4567-9fcc-8e0dca75c319" containerName="authorino" Nov 01 06:17:40.654333 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:40.654212 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c6a47db-4dcc-4314-8926-a898ff1ce943" containerName="authorino" Nov 01 06:17:40.657189 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:40.657175 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-446vb" Nov 01 06:17:40.710463 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:40.710430 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-jf8sg\"" Nov 01 06:17:40.713540 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:40.713505 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-446vb"] Nov 01 06:17:40.747561 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:40.747532 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8m6h\" (UniqueName: \"kubernetes.io/projected/57957a77-1fea-4f3d-a619-a655a33498be-kube-api-access-t8m6h\") pod \"llmisvc-controller-manager-6466c554d8-446vb\" (UID: \"57957a77-1fea-4f3d-a619-a655a33498be\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-446vb" Nov 01 06:17:40.747734 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:40.747604 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57957a77-1fea-4f3d-a619-a655a33498be-cert\") pod \"llmisvc-controller-manager-6466c554d8-446vb\" (UID: \"57957a77-1fea-4f3d-a619-a655a33498be\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-446vb" Nov 01 06:17:40.810253 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:40.810224 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 01 06:17:40.848712 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:40.848666 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8m6h\" (UniqueName: \"kubernetes.io/projected/57957a77-1fea-4f3d-a619-a655a33498be-kube-api-access-t8m6h\") pod \"llmisvc-controller-manager-6466c554d8-446vb\" (UID: \"57957a77-1fea-4f3d-a619-a655a33498be\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-446vb" Nov 01 06:17:40.848948 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:40.848747 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57957a77-1fea-4f3d-a619-a655a33498be-cert\") pod \"llmisvc-controller-manager-6466c554d8-446vb\" (UID: \"57957a77-1fea-4f3d-a619-a655a33498be\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-446vb" Nov 01 06:17:40.851488 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:40.851463 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57957a77-1fea-4f3d-a619-a655a33498be-cert\") pod \"llmisvc-controller-manager-6466c554d8-446vb\" (UID: \"57957a77-1fea-4f3d-a619-a655a33498be\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-446vb" Nov 01 06:17:40.905694 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:40.905618 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8m6h\" (UniqueName: \"kubernetes.io/projected/57957a77-1fea-4f3d-a619-a655a33498be-kube-api-access-t8m6h\") pod \"llmisvc-controller-manager-6466c554d8-446vb\" (UID: \"57957a77-1fea-4f3d-a619-a655a33498be\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-446vb" Nov 01 06:17:40.969309 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:40.969273 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-446vb" Nov 01 06:17:41.337506 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:41.337469 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-446vb"] Nov 01 06:17:41.338800 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:17:41.338777 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57957a77_1fea_4f3d_a619_a655a33498be.slice/crio-556d3bb85d2b893a64845565814f7aa373e3bd203c7a51a31fa40ad3f6732169 WatchSource:0}: Error finding container 556d3bb85d2b893a64845565814f7aa373e3bd203c7a51a31fa40ad3f6732169: Status 404 returned error can't find the container with id 556d3bb85d2b893a64845565814f7aa373e3bd203c7a51a31fa40ad3f6732169 Nov 01 06:17:42.187242 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:42.187209 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-446vb" event={"ID":"57957a77-1fea-4f3d-a619-a655a33498be","Type":"ContainerStarted","Data":"556d3bb85d2b893a64845565814f7aa373e3bd203c7a51a31fa40ad3f6732169"} Nov 01 06:17:44.204043 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:44.204000 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-446vb" event={"ID":"57957a77-1fea-4f3d-a619-a655a33498be","Type":"ContainerStarted","Data":"62e5b9660a430de53057cf8ddf5a94338dba511fbfed66dc5655a16683daa84d"} Nov 01 06:17:44.204499 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:44.204200 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-446vb" Nov 01 06:17:44.317518 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:44.317462 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-446vb" podStartSLOduration=2.039252622 podStartE2EDuration="4.31744581s" totalImagesPullingTime="2.278193188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:17:44.315613559 +0000 UTC m=+474.079413416" watchObservedRunningTime="2026-11-01 06:17:44.31744581 +0000 UTC m=+474.081245704" Nov 01 06:17:52.359980 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:17:52.359945 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:18:06.778395 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:18:06.778316 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:18:15.212294 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:18:15.212264 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-446vb" Nov 01 06:18:16.466269 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:18:16.466226 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:18:16.750515 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:18:16.750422 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:18:22.789967 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:18:22.789931 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:18:22.982102 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:18:22.982066 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:18:36.778524 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:18:36.778488 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:18:36.914333 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:18:36.914295 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:18:45.478129 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:18:45.478095 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:18:45.976459 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:18:45.976413 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:18:47.214804 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:18:47.214772 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:18:47.358151 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:18:47.358113 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:18:55.480277 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:18:55.480240 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:19:03.627526 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:03.627489 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:19:04.285821 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:04.285785 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:19:04.478227 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:04.478190 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:19:06.782182 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:19:06.782140 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:19:06.933986 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:06.933949 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:19:10.796862 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:10.796799 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:19:14.734048 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:14.734002 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:19:18.712615 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:18.712580 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:19:21.976750 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:21.976710 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:19:25.570853 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:25.570790 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:19:36.779441 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:19:36.779352 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:19:37.212191 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:37.212146 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:19:54.393468 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:54.393436 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58588b67c6-mr9nn"] Nov 01 06:19:54.396868 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:54.396850 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58588b67c6-mr9nn" Nov 01 06:19:54.451180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:54.451147 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58588b67c6-mr9nn"] Nov 01 06:19:54.529257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:54.529226 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/413fadb6-87d9-4270-b53e-a7bf64b32d60-tls-cert\") pod \"authorino-58588b67c6-mr9nn\" (UID: \"413fadb6-87d9-4270-b53e-a7bf64b32d60\") " pod="kuadrant-system/authorino-58588b67c6-mr9nn" Nov 01 06:19:54.529448 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:54.529286 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4wpr\" (UniqueName: \"kubernetes.io/projected/413fadb6-87d9-4270-b53e-a7bf64b32d60-kube-api-access-w4wpr\") pod \"authorino-58588b67c6-mr9nn\" (UID: \"413fadb6-87d9-4270-b53e-a7bf64b32d60\") " pod="kuadrant-system/authorino-58588b67c6-mr9nn" Nov 01 06:19:54.630301 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:54.630262 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/413fadb6-87d9-4270-b53e-a7bf64b32d60-tls-cert\") pod \"authorino-58588b67c6-mr9nn\" (UID: \"413fadb6-87d9-4270-b53e-a7bf64b32d60\") " pod="kuadrant-system/authorino-58588b67c6-mr9nn" Nov 01 06:19:54.630476 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:54.630329 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4wpr\" (UniqueName: \"kubernetes.io/projected/413fadb6-87d9-4270-b53e-a7bf64b32d60-kube-api-access-w4wpr\") pod \"authorino-58588b67c6-mr9nn\" (UID: \"413fadb6-87d9-4270-b53e-a7bf64b32d60\") " pod="kuadrant-system/authorino-58588b67c6-mr9nn" Nov 01 06:19:54.633048 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:54.633023 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/413fadb6-87d9-4270-b53e-a7bf64b32d60-tls-cert\") pod \"authorino-58588b67c6-mr9nn\" (UID: \"413fadb6-87d9-4270-b53e-a7bf64b32d60\") " pod="kuadrant-system/authorino-58588b67c6-mr9nn" Nov 01 06:19:54.688035 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:54.687957 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4wpr\" (UniqueName: \"kubernetes.io/projected/413fadb6-87d9-4270-b53e-a7bf64b32d60-kube-api-access-w4wpr\") pod \"authorino-58588b67c6-mr9nn\" (UID: \"413fadb6-87d9-4270-b53e-a7bf64b32d60\") " pod="kuadrant-system/authorino-58588b67c6-mr9nn" Nov 01 06:19:54.708565 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:54.708537 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58588b67c6-mr9nn" Nov 01 06:19:55.072519 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:55.072425 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:19:55.084800 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:55.084775 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58588b67c6-mr9nn"] Nov 01 06:19:55.086057 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:19:55.086018 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod413fadb6_87d9_4270_b53e_a7bf64b32d60.slice/crio-90fca5f510bc1cd38267e22e740d0fc3c066545c490a7e5c563f209ff688f3c5 WatchSource:0}: Error finding container 90fca5f510bc1cd38267e22e740d0fc3c066545c490a7e5c563f209ff688f3c5: Status 404 returned error can't find the container with id 90fca5f510bc1cd38267e22e740d0fc3c066545c490a7e5c563f209ff688f3c5 Nov 01 06:19:55.223504 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:55.223471 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:19:55.930886 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:55.930820 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58588b67c6-mr9nn" event={"ID":"413fadb6-87d9-4270-b53e-a7bf64b32d60","Type":"ContainerStarted","Data":"90fca5f510bc1cd38267e22e740d0fc3c066545c490a7e5c563f209ff688f3c5"} Nov 01 06:19:56.572243 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:56.572204 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:19:56.724694 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:56.724666 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:19:56.937691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:56.937659 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58588b67c6-mr9nn" event={"ID":"413fadb6-87d9-4270-b53e-a7bf64b32d60","Type":"ContainerStarted","Data":"71c3412d92973435b85e2efe4cd0081f6cd8bb0cd1763900561fe426ac59e6cf"} Nov 01 06:19:57.054756 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:57.054708 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-58588b67c6-mr9nn" podStartSLOduration=1.476520992 podStartE2EDuration="3.054692987s" totalImagesPullingTime="1.578171995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:57.052767447 +0000 UTC m=+606.816567306" watchObservedRunningTime="2026-11-01 06:19:57.054692987 +0000 UTC m=+606.818492843" Nov 01 06:19:57.106423 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:57.106386 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64c8bb8959-2zlbz"] Nov 01 06:19:57.106604 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:57.106576 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-64c8bb8959-2zlbz" podUID="e4749577-adda-47e0-b660-8f12cf0f08fb" containerName="authorino" containerID="cri-o://951407e4fbb8347037216777125f088470e284706ba299942e0874508b184c6e" gracePeriod=30 Nov 01 06:19:57.365676 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:57.365648 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64c8bb8959-2zlbz" Nov 01 06:19:57.457259 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:57.457186 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4749577-adda-47e0-b660-8f12cf0f08fb-tls-cert\" (UniqueName: \"kubernetes.io/secret/e4749577-adda-47e0-b660-8f12cf0f08fb-tls-cert\") pod \"e4749577-adda-47e0-b660-8f12cf0f08fb\" (UID: \"e4749577-adda-47e0-b660-8f12cf0f08fb\") " Nov 01 06:19:57.457451 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:57.457304 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4749577-adda-47e0-b660-8f12cf0f08fb-kube-api-access-kgsf8\" (UniqueName: \"kubernetes.io/projected/e4749577-adda-47e0-b660-8f12cf0f08fb-kube-api-access-kgsf8\") pod \"e4749577-adda-47e0-b660-8f12cf0f08fb\" (UID: \"e4749577-adda-47e0-b660-8f12cf0f08fb\") " Nov 01 06:19:57.460148 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:57.460111 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4749577-adda-47e0-b660-8f12cf0f08fb-kube-api-access-kgsf8" pod "e4749577-adda-47e0-b660-8f12cf0f08fb" (UID: "e4749577-adda-47e0-b660-8f12cf0f08fb"). InnerVolumeSpecName "kube-api-access-kgsf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:19:57.471262 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:57.471229 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4749577-adda-47e0-b660-8f12cf0f08fb-tls-cert" pod "e4749577-adda-47e0-b660-8f12cf0f08fb" (UID: "e4749577-adda-47e0-b660-8f12cf0f08fb"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:19:57.558396 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:57.558311 4385 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e4749577-adda-47e0-b660-8f12cf0f08fb-tls-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:19:57.558396 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:57.558344 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgsf8\" (UniqueName: \"kubernetes.io/projected/e4749577-adda-47e0-b660-8f12cf0f08fb-kube-api-access-kgsf8\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:19:57.952441 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:57.952398 4385 generic.go:356] "Generic (PLEG): container finished" podID="e4749577-adda-47e0-b660-8f12cf0f08fb" containerID="951407e4fbb8347037216777125f088470e284706ba299942e0874508b184c6e" exitCode=0 Nov 01 06:19:57.952980 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:57.952460 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64c8bb8959-2zlbz" Nov 01 06:19:57.952980 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:57.952505 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64c8bb8959-2zlbz" event={"ID":"e4749577-adda-47e0-b660-8f12cf0f08fb","Type":"ContainerDied","Data":"951407e4fbb8347037216777125f088470e284706ba299942e0874508b184c6e"} Nov 01 06:19:57.952980 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:57.952580 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64c8bb8959-2zlbz" event={"ID":"e4749577-adda-47e0-b660-8f12cf0f08fb","Type":"ContainerDied","Data":"26d3bb08156591e4ef17d123be1371ba55d379e40776c9772027db17f548c9c4"} Nov 01 06:19:57.953222 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:57.952602 4385 scope.go:122] "RemoveContainer" containerID="951407e4fbb8347037216777125f088470e284706ba299942e0874508b184c6e" Nov 01 06:19:57.963325 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:57.963288 4385 scope.go:122] "RemoveContainer" containerID="951407e4fbb8347037216777125f088470e284706ba299942e0874508b184c6e" Nov 01 06:19:57.963634 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:19:57.963601 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"951407e4fbb8347037216777125f088470e284706ba299942e0874508b184c6e\": container with ID starting with 951407e4fbb8347037216777125f088470e284706ba299942e0874508b184c6e not found: ID does not exist" containerID="951407e4fbb8347037216777125f088470e284706ba299942e0874508b184c6e" Nov 01 06:19:57.963765 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:57.963635 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"951407e4fbb8347037216777125f088470e284706ba299942e0874508b184c6e"} err="failed to get container status \"951407e4fbb8347037216777125f088470e284706ba299942e0874508b184c6e\": rpc error: code = NotFound desc = could not find container \"951407e4fbb8347037216777125f088470e284706ba299942e0874508b184c6e\": container with ID starting with 951407e4fbb8347037216777125f088470e284706ba299942e0874508b184c6e not found: ID does not exist" Nov 01 06:19:58.120529 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:58.120496 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64c8bb8959-2zlbz"] Nov 01 06:19:58.129993 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:58.129967 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-64c8bb8959-2zlbz"] Nov 01 06:19:58.384023 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:19:58.383987 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4749577-adda-47e0-b660-8f12cf0f08fb" path="/var/lib/kubelet/pods/e4749577-adda-47e0-b660-8f12cf0f08fb/volumes" Nov 01 06:20:06.779963 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:20:06.779912 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:20:20.464220 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:20.464189 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:23.668305 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:23.668272 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:23.811281 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:23.811244 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:24.237430 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:24.237396 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:24.487551 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:24.487429 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:24.635854 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:24.635802 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:24.835387 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:24.835342 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:24.989882 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:24.989849 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:25.140704 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:25.140616 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:27.338953 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:27.338905 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:27.530595 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:27.530559 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:28.203340 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:28.203305 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:28.443746 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:28.443708 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:28.592112 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:28.592081 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:29.045366 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:29.045286 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:29.220881 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:29.220815 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:29.378751 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:29.378710 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:30.511533 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:30.511500 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:30.853032 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:30.852995 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:31.065086 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:31.065044 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:31.420906 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:31.420871 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:31.581885 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:31.581825 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:34.673425 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:34.673390 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:34.866422 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:34.866390 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:35.014579 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:35.014483 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:36.110032 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:36.109997 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:36.291344 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:36.291302 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:36.778958 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:20:36.778924 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:20:37.569959 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:37.569916 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:38.527431 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:38.527399 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:38.691114 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:38.691077 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:38.902456 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:38.902421 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww"] Nov 01 06:20:38.902804 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:38.902792 4385 cpu_manager.go:414] "containerMap: removing container" podUID="e4749577-adda-47e0-b660-8f12cf0f08fb" containerName="authorino" Nov 01 06:20:38.902892 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:38.902805 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4749577-adda-47e0-b660-8f12cf0f08fb" containerName="authorino" Nov 01 06:20:38.902933 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:38.902907 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4749577-adda-47e0-b660-8f12cf0f08fb" containerName="authorino" Nov 01 06:20:38.906051 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:38.906028 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww" Nov 01 06:20:38.960354 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:38.960323 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-3ba2f3\"" Nov 01 06:20:39.009725 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:39.009697 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-3ba2f3-dockercfg-vl87v\"" Nov 01 06:20:39.012643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:39.012620 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww"] Nov 01 06:20:39.033250 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:39.033219 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnfxg\" (UniqueName: \"kubernetes.io/projected/d16e3ebb-dc72-474e-b24c-8f92b7335140-kube-api-access-jnfxg\") pod \"payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww\" (UID: \"d16e3ebb-dc72-474e-b24c-8f92b7335140\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww" Nov 01 06:20:39.033413 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:39.033258 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d16e3ebb-dc72-474e-b24c-8f92b7335140-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww\" (UID: \"d16e3ebb-dc72-474e-b24c-8f92b7335140\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww" Nov 01 06:20:39.133773 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:39.133742 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnfxg\" (UniqueName: \"kubernetes.io/projected/d16e3ebb-dc72-474e-b24c-8f92b7335140-kube-api-access-jnfxg\") pod \"payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww\" (UID: \"d16e3ebb-dc72-474e-b24c-8f92b7335140\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww" Nov 01 06:20:39.134021 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:39.133785 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d16e3ebb-dc72-474e-b24c-8f92b7335140-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww\" (UID: \"d16e3ebb-dc72-474e-b24c-8f92b7335140\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww" Nov 01 06:20:39.134481 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:39.134464 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d16e3ebb-dc72-474e-b24c-8f92b7335140-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww\" (UID: \"d16e3ebb-dc72-474e-b24c-8f92b7335140\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww" Nov 01 06:20:39.193195 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:39.193119 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnfxg\" (UniqueName: \"kubernetes.io/projected/d16e3ebb-dc72-474e-b24c-8f92b7335140-kube-api-access-jnfxg\") pod \"payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww\" (UID: \"d16e3ebb-dc72-474e-b24c-8f92b7335140\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww" Nov 01 06:20:39.218895 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:39.218866 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww" Nov 01 06:20:39.498365 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:39.498272 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:39.594362 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:39.594333 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww"] Nov 01 06:20:39.595563 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:20:39.595541 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16e3ebb_dc72_474e_b24c_8f92b7335140.slice/crio-3a84783b07ede293e364dcb0b729e3b6af5af469ef1089ab6748ba98da518859 WatchSource:0}: Error finding container 3a84783b07ede293e364dcb0b729e3b6af5af469ef1089ab6748ba98da518859: Status 404 returned error can't find the container with id 3a84783b07ede293e364dcb0b729e3b6af5af469ef1089ab6748ba98da518859 Nov 01 06:20:39.597986 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:39.597970 4385 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:20:39.661732 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:39.661699 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:39.807753 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:39.807653 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:40.179306 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:40.179267 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww" event={"ID":"d16e3ebb-dc72-474e-b24c-8f92b7335140","Type":"ContainerStarted","Data":"3a84783b07ede293e364dcb0b729e3b6af5af469ef1089ab6748ba98da518859"} Nov 01 06:20:43.200443 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:43.200407 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww" event={"ID":"d16e3ebb-dc72-474e-b24c-8f92b7335140","Type":"ContainerStarted","Data":"b5bda7d6d39714fe2eab2a859eff84019fd2a7197576ad787027fc800bf8cad2"} Nov 01 06:20:43.200884 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:43.200570 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww" Nov 01 06:20:43.314331 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:43.314271 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww" podStartSLOduration=2.78764902 podStartE2EDuration="5.314254579s" totalImagesPullingTime="2.526605559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:43.313935151 +0000 UTC m=+653.077735008" watchObservedRunningTime="2026-11-01 06:20:43.314254579 +0000 UTC m=+653.078054438" Nov 01 06:20:44.281083 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:44.281039 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:44.437650 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:44.437619 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:44.601738 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:44.601704 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:49.131938 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:49.131901 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:53.964634 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:53.964596 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:20:54.210961 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:54.210932 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww" Nov 01 06:20:58.737804 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:20:58.737772 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:03.377371 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:03.377286 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:03.768748 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:03.768659 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:03.980296 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:03.980249 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:04.137997 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:04.137959 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:06.781486 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:21:06.781451 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:21:07.126315 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.126276 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd"] Nov 01 06:21:07.129702 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.129681 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:07.180727 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.180692 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-model-cache\") pod \"simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:07.180919 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.180762 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-tmp-dir\") pod \"simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:07.180980 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.180918 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04ae8d7a-c300-4ddc-89a2-7d3400685bda-tls-certs\") pod \"simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:07.180980 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.180964 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-home\") pod \"simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:07.181096 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.181075 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-dshm\") pod \"simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:07.181166 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.181151 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6svl5\" (UniqueName: \"kubernetes.io/projected/04ae8d7a-c300-4ddc-89a2-7d3400685bda-kube-api-access-6svl5\") pod \"simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:07.232208 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.232174 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3ba2f3\"/\"default-dockercfg-nhmkx\"" Nov 01 06:21:07.232208 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.232173 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3ba2f3\"/\"openshift-service-ca.crt\"" Nov 01 06:21:07.232470 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.232241 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3ba2f3\"/\"kube-root-ca.crt\"" Nov 01 06:21:07.232537 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.232479 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3ba2f3\"/\"simulator-w0-3ba2f3-kserve-self-signed-certs\"" Nov 01 06:21:07.233401 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.233360 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:07.235158 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.235141 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd"] Nov 01 06:21:07.282637 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.282604 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-model-cache\") pod \"simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:07.282867 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.282686 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-tmp-dir\") pod \"simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:07.282867 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.282733 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04ae8d7a-c300-4ddc-89a2-7d3400685bda-tls-certs\") pod \"simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:07.282867 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.282761 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-home\") pod \"simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:07.282867 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.282791 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-dshm\") pod \"simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:07.283105 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.282881 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6svl5\" (UniqueName: \"kubernetes.io/projected/04ae8d7a-c300-4ddc-89a2-7d3400685bda-kube-api-access-6svl5\") pod \"simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:07.283162 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.283098 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-model-cache\") pod \"simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:07.283453 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.283435 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-tmp-dir\") pod \"simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:07.283591 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.283483 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-home\") pod \"simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:07.285483 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.285465 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-dshm\") pod \"simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:07.285984 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.285961 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04ae8d7a-c300-4ddc-89a2-7d3400685bda-tls-certs\") pod \"simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:07.340533 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.340498 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6svl5\" (UniqueName: \"kubernetes.io/projected/04ae8d7a-c300-4ddc-89a2-7d3400685bda-kube-api-access-6svl5\") pod \"simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:07.441889 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.441760 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:07.820293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:07.820267 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd"] Nov 01 06:21:07.821326 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:21:07.821302 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04ae8d7a_c300_4ddc_89a2_7d3400685bda.slice/crio-12d95a6042ac72f5538caf34d21a513c6784a6d834f256df9ff011d7ef1f4ee6 WatchSource:0}: Error finding container 12d95a6042ac72f5538caf34d21a513c6784a6d834f256df9ff011d7ef1f4ee6: Status 404 returned error can't find the container with id 12d95a6042ac72f5538caf34d21a513c6784a6d834f256df9ff011d7ef1f4ee6 Nov 01 06:21:08.343280 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:08.343243 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" event={"ID":"04ae8d7a-c300-4ddc-89a2-7d3400685bda","Type":"ContainerStarted","Data":"12d95a6042ac72f5538caf34d21a513c6784a6d834f256df9ff011d7ef1f4ee6"} Nov 01 06:21:08.417680 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:08.417650 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:08.818248 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:08.818175 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:08.990676 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:08.990412 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:09.143706 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:09.143673 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:09.561197 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:09.561112 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:10.135732 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:10.135696 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:10.341594 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:10.341540 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:10.361360 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:10.361317 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" event={"ID":"04ae8d7a-c300-4ddc-89a2-7d3400685bda","Type":"ContainerStarted","Data":"b51383f646a9128fecd1c27e0f164ceeb92e376a112ca241bc8151b7cb053854"} Nov 01 06:21:10.442945 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:10.442910 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:10.492531 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:10.492479 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" podStartSLOduration=1.328436498 podStartE2EDuration="3.492466578s" totalImagesPullingTime="2.16403008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:21:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:21:10.489737265 +0000 UTC m=+680.253537123" watchObservedRunningTime="2026-11-01 06:21:10.492466578 +0000 UTC m=+680.256266436" Nov 01 06:21:10.588951 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:10.588914 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:11.182987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:11.182938 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:11.374500 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:11.374463 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:13.947415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:13.947374 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:15.610537 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:15.610501 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:15.815917 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:15.815876 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:16.003218 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:16.003128 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:17.442732 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:17.442706 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:17.455649 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:17.455618 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:18.414920 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:18.414896 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 06:21:18.793972 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:18.793883 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:18.937728 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:18.937687 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:19.083296 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.083261 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:19.182245 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.182214 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx"] Nov 01 06:21:19.189616 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.189588 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx" Nov 01 06:21:19.244081 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.244047 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx"] Nov 01 06:21:19.288853 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.288808 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-85584b-dockercfg-frljn\"" Nov 01 06:21:19.288853 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.288845 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-85584b\"" Nov 01 06:21:19.304080 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.304052 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hxvx\" (UniqueName: \"kubernetes.io/projected/0213cfe7-3498-4570-8de5-1bb46d87503c-kube-api-access-2hxvx\") pod \"payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx\" (UID: \"0213cfe7-3498-4570-8de5-1bb46d87503c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx" Nov 01 06:21:19.304213 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.304105 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0213cfe7-3498-4570-8de5-1bb46d87503c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx\" (UID: \"0213cfe7-3498-4570-8de5-1bb46d87503c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx" Nov 01 06:21:19.343477 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.343406 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466"] Nov 01 06:21:19.346997 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.346978 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466" Nov 01 06:21:19.402280 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.402252 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466"] Nov 01 06:21:19.405160 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.405131 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0213cfe7-3498-4570-8de5-1bb46d87503c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx\" (UID: \"0213cfe7-3498-4570-8de5-1bb46d87503c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx" Nov 01 06:21:19.406022 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.405232 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hxvx\" (UniqueName: \"kubernetes.io/projected/0213cfe7-3498-4570-8de5-1bb46d87503c-kube-api-access-2hxvx\") pod \"payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx\" (UID: \"0213cfe7-3498-4570-8de5-1bb46d87503c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx" Nov 01 06:21:19.406246 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.406223 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0213cfe7-3498-4570-8de5-1bb46d87503c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx\" (UID: \"0213cfe7-3498-4570-8de5-1bb46d87503c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx" Nov 01 06:21:19.464146 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.464120 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hxvx\" (UniqueName: \"kubernetes.io/projected/0213cfe7-3498-4570-8de5-1bb46d87503c-kube-api-access-2hxvx\") pod \"payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx\" (UID: \"0213cfe7-3498-4570-8de5-1bb46d87503c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx" Nov 01 06:21:19.503752 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.503730 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx" Nov 01 06:21:19.507913 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.507105 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ea0a3f-5ef6-45fe-ba2a-894a232e81b4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466\" (UID: \"48ea0a3f-5ef6-45fe-ba2a-894a232e81b4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466" Nov 01 06:21:19.507913 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.507237 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftx2f\" (UniqueName: \"kubernetes.io/projected/48ea0a3f-5ef6-45fe-ba2a-894a232e81b4-kube-api-access-ftx2f\") pod \"payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466\" (UID: \"48ea0a3f-5ef6-45fe-ba2a-894a232e81b4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466" Nov 01 06:21:19.608463 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.608392 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ea0a3f-5ef6-45fe-ba2a-894a232e81b4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466\" (UID: \"48ea0a3f-5ef6-45fe-ba2a-894a232e81b4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466" Nov 01 06:21:19.608463 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.608441 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftx2f\" (UniqueName: \"kubernetes.io/projected/48ea0a3f-5ef6-45fe-ba2a-894a232e81b4-kube-api-access-ftx2f\") pod \"payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466\" (UID: \"48ea0a3f-5ef6-45fe-ba2a-894a232e81b4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466" Nov 01 06:21:19.609154 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.609130 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ea0a3f-5ef6-45fe-ba2a-894a232e81b4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466\" (UID: \"48ea0a3f-5ef6-45fe-ba2a-894a232e81b4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466" Nov 01 06:21:19.665086 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.665062 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftx2f\" (UniqueName: \"kubernetes.io/projected/48ea0a3f-5ef6-45fe-ba2a-894a232e81b4-kube-api-access-ftx2f\") pod \"payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466\" (UID: \"48ea0a3f-5ef6-45fe-ba2a-894a232e81b4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466" Nov 01 06:21:19.874286 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.874253 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx"] Nov 01 06:21:19.875138 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:21:19.875109 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0213cfe7_3498_4570_8de5_1bb46d87503c.slice/crio-f8eeb294360f430df44c5837cf0f4701b8e8e16be5b3e856daa968cfeb414dba WatchSource:0}: Error finding container f8eeb294360f430df44c5837cf0f4701b8e8e16be5b3e856daa968cfeb414dba: Status 404 returned error can't find the container with id f8eeb294360f430df44c5837cf0f4701b8e8e16be5b3e856daa968cfeb414dba Nov 01 06:21:19.960020 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:19.959998 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466" Nov 01 06:21:20.330114 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:20.329915 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466"] Nov 01 06:21:20.332619 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:21:20.332595 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ea0a3f_5ef6_45fe_ba2a_894a232e81b4.slice/crio-35e4319eae2c297d03dfc678345a86c6a97092928c407c9112ab0ee839729d86 WatchSource:0}: Error finding container 35e4319eae2c297d03dfc678345a86c6a97092928c407c9112ab0ee839729d86: Status 404 returned error can't find the container with id 35e4319eae2c297d03dfc678345a86c6a97092928c407c9112ab0ee839729d86 Nov 01 06:21:20.417968 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:20.417934 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466" event={"ID":"48ea0a3f-5ef6-45fe-ba2a-894a232e81b4","Type":"ContainerStarted","Data":"0d81d8ae9a97572d6a10094e28af5a4ef7ef31ee7c2640f1032a103695ec49de"} Nov 01 06:21:20.418151 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:20.417997 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466" Nov 01 06:21:20.418151 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:20.418020 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466" event={"ID":"48ea0a3f-5ef6-45fe-ba2a-894a232e81b4","Type":"ContainerStarted","Data":"35e4319eae2c297d03dfc678345a86c6a97092928c407c9112ab0ee839729d86"} Nov 01 06:21:20.419624 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:20.419597 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx" event={"ID":"0213cfe7-3498-4570-8de5-1bb46d87503c","Type":"ContainerStarted","Data":"11a3256fac39f47cd3fc52fa388267e2cb70cf02219fb565ecd5648c2cfcbf2a"} Nov 01 06:21:20.419749 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:20.419647 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx" event={"ID":"0213cfe7-3498-4570-8de5-1bb46d87503c","Type":"ContainerStarted","Data":"f8eeb294360f430df44c5837cf0f4701b8e8e16be5b3e856daa968cfeb414dba"} Nov 01 06:21:20.419749 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:20.419700 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx" Nov 01 06:21:20.532460 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:20.532402 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466" podStartSLOduration=1.532382534 podStartE2EDuration="1.532382534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:21:20.529294985 +0000 UTC m=+690.293094843" watchObservedRunningTime="2026-11-01 06:21:20.532382534 +0000 UTC m=+690.296182392" Nov 01 06:21:20.644523 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:20.644425 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx" podStartSLOduration=1.6444114239999998 podStartE2EDuration="1.644411424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:21:20.644014188 +0000 UTC m=+690.407814047" watchObservedRunningTime="2026-11-01 06:21:20.644411424 +0000 UTC m=+690.408211280" Nov 01 06:21:23.624744 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:23.624709 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:28.314366 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:28.314332 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:30.335793 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.335761 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2"] Nov 01 06:21:30.339185 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.339170 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.511691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.511656 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.511691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.511696 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/753c422d-48cc-40ee-8673-208ed65fd311-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.511905 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.511713 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcftq\" (UniqueName: \"kubernetes.io/projected/753c422d-48cc-40ee-8673-208ed65fd311-kube-api-access-dcftq\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.511905 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.511827 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.511905 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.511874 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.512050 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.511913 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/753c422d-48cc-40ee-8673-208ed65fd311-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.512050 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.512014 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.512142 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.512059 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.512142 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.512116 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/753c422d-48cc-40ee-8673-208ed65fd311-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.540047 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.540022 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2"] Nov 01 06:21:30.612636 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.612543 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.612636 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.612594 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.612915 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.612648 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/753c422d-48cc-40ee-8673-208ed65fd311-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.612915 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.612672 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.612915 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.612700 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.612915 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.612724 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/753c422d-48cc-40ee-8673-208ed65fd311-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.612915 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.612767 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.612915 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.612802 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/753c422d-48cc-40ee-8673-208ed65fd311-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.612915 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.612827 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcftq\" (UniqueName: \"kubernetes.io/projected/753c422d-48cc-40ee-8673-208ed65fd311-kube-api-access-dcftq\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.613298 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.613150 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.613359 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.613331 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.613423 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.613381 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.613483 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.613459 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/753c422d-48cc-40ee-8673-208ed65fd311-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.613483 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.613469 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.615026 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.615006 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.615288 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.615269 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/753c422d-48cc-40ee-8673-208ed65fd311-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.670603 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.670573 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/753c422d-48cc-40ee-8673-208ed65fd311-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.670749 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.670640 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcftq\" (UniqueName: \"kubernetes.io/projected/753c422d-48cc-40ee-8673-208ed65fd311-kube-api-access-dcftq\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r5wn2\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:30.951960 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:30.951885 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:31.377084 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:31.377050 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2"] Nov 01 06:21:31.415054 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:21:31.415029 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod753c422d_48cc_40ee_8673_208ed65fd311.slice/crio-1279c3c772decb9828e8bb97a9198bd25553a8a32f224178afcda6a1b581cde3 WatchSource:0}: Error finding container 1279c3c772decb9828e8bb97a9198bd25553a8a32f224178afcda6a1b581cde3: Status 404 returned error can't find the container with id 1279c3c772decb9828e8bb97a9198bd25553a8a32f224178afcda6a1b581cde3 Nov 01 06:21:31.417480 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:31.417448 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:21:31.417575 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:31.417512 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:21:31.417575 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:31.417543 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:21:31.428024 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:31.428002 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx" Nov 01 06:21:31.428143 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:31.428068 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466" Nov 01 06:21:31.480965 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:31.480938 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" event={"ID":"753c422d-48cc-40ee-8673-208ed65fd311","Type":"ContainerStarted","Data":"1279c3c772decb9828e8bb97a9198bd25553a8a32f224178afcda6a1b581cde3"} Nov 01 06:21:32.048579 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.048539 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r"] Nov 01 06:21:32.052659 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.052639 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:32.106480 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.106441 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8ff537\"/\"kube-root-ca.crt\"" Nov 01 06:21:32.109509 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.109481 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r"] Nov 01 06:21:32.149311 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.149269 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:32.155913 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.155824 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8ff537\"/\"default-dockercfg-hl7g2\"" Nov 01 06:21:32.155913 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.155824 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8ff537\"/\"openshift-service-ca.crt\"" Nov 01 06:21:32.156104 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.155923 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8ff537\"/\"simulator-w2-8ff537-kserve-self-signed-certs\"" Nov 01 06:21:32.225921 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.225884 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-home\") pod \"simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:32.225921 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.225929 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-model-cache\") pod \"simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:32.226171 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.225954 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfmxs\" (UniqueName: \"kubernetes.io/projected/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-kube-api-access-jfmxs\") pod \"simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:32.226171 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.226071 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-tls-certs\") pod \"simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:32.226252 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.226178 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-dshm\") pod \"simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:32.226293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.226257 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-tmp-dir\") pod \"simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:32.327643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.327603 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-dshm\") pod \"simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:32.327863 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.327660 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-tmp-dir\") pod \"simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:32.327863 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.327719 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-home\") pod \"simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:32.327863 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.327742 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-model-cache\") pod \"simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:32.327863 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.327767 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfmxs\" (UniqueName: \"kubernetes.io/projected/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-kube-api-access-jfmxs\") pod \"simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:32.327863 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.327805 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-tls-certs\") pod \"simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:32.328272 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.328245 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-home\") pod \"simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:32.328420 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.328394 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-model-cache\") pod \"simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:32.328515 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.328493 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-tmp-dir\") pod \"simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:32.330600 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.330573 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-dshm\") pod \"simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:32.331063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.331043 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-tls-certs\") pod \"simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:32.386767 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.386691 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfmxs\" (UniqueName: \"kubernetes.io/projected/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-kube-api-access-jfmxs\") pod \"simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:32.506794 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.506738 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" event={"ID":"753c422d-48cc-40ee-8673-208ed65fd311","Type":"ContainerStarted","Data":"bec44a5ed80eb8f4f920f6045eb96c59b456fd20f626e5b4ea45d3b3187beb28"} Nov 01 06:21:32.623086 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.622980 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" podStartSLOduration=2.622965455 podStartE2EDuration="2.622965455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:21:32.622657633 +0000 UTC m=+702.386457491" watchObservedRunningTime="2026-11-01 06:21:32.622965455 +0000 UTC m=+702.386765311" Nov 01 06:21:32.665954 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.665917 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:32.952387 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.952315 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:32.953738 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.953714 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-r5wn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 06:21:32.953923 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.953760 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" podUID="753c422d-48cc-40ee-8673-208ed65fd311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 06:21:32.994856 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:32.994510 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:33.063689 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:33.063658 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r"] Nov 01 06:21:33.065501 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:21:33.065471 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf35d3f2c_2ea5_47d6_b808_93a12e4923c0.slice/crio-1774dd11e0e6ef26c9f13d5386f3b96ef04e6903e0f5ac23dc5ff621e016cb3b WatchSource:0}: Error finding container 1774dd11e0e6ef26c9f13d5386f3b96ef04e6903e0f5ac23dc5ff621e016cb3b: Status 404 returned error can't find the container with id 1774dd11e0e6ef26c9f13d5386f3b96ef04e6903e0f5ac23dc5ff621e016cb3b Nov 01 06:21:33.136514 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:33.136477 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:33.323135 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:33.323052 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:33.513809 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:33.513769 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" event={"ID":"f35d3f2c-2ea5-47d6-b808-93a12e4923c0","Type":"ContainerStarted","Data":"e1b0edc39f77de1dff156722ed945610c51b04dbbc527d871269363a981eac64"} Nov 01 06:21:33.513809 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:33.513809 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" event={"ID":"f35d3f2c-2ea5-47d6-b808-93a12e4923c0","Type":"ContainerStarted","Data":"1774dd11e0e6ef26c9f13d5386f3b96ef04e6903e0f5ac23dc5ff621e016cb3b"} Nov 01 06:21:33.631028 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:33.630967 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" podStartSLOduration=2.630951301 podStartE2EDuration="2.630951301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:21:33.62837942 +0000 UTC m=+703.392179273" watchObservedRunningTime="2026-11-01 06:21:33.630951301 +0000 UTC m=+703.394751160" Nov 01 06:21:33.666614 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:33.666574 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:33.952473 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:33.952389 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-r5wn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 06:21:33.952473 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:33.952449 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" podUID="753c422d-48cc-40ee-8673-208ed65fd311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 06:21:34.952379 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:34.952338 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-r5wn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 06:21:34.952776 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:34.952395 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" podUID="753c422d-48cc-40ee-8673-208ed65fd311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 06:21:35.656392 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.656357 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx"] Nov 01 06:21:35.660022 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.659999 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:35.759534 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.759496 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6571db97-e767-4917-9543-5ebd4e6ff1e5-tls-certs\") pod \"simulator-w4-190724-kserve-5f758995c6-nz6xx\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:35.759728 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.759553 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-home\") pod \"simulator-w4-190724-kserve-5f758995c6-nz6xx\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:35.759728 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.759574 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-dshm\") pod \"simulator-w4-190724-kserve-5f758995c6-nz6xx\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:35.759828 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.759760 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-model-cache\") pod \"simulator-w4-190724-kserve-5f758995c6-nz6xx\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:35.759828 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.759797 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-tmp-dir\") pod \"simulator-w4-190724-kserve-5f758995c6-nz6xx\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:35.759828 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.759814 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mvpj\" (UniqueName: \"kubernetes.io/projected/6571db97-e767-4917-9543-5ebd4e6ff1e5-kube-api-access-4mvpj\") pod \"simulator-w4-190724-kserve-5f758995c6-nz6xx\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:35.762712 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.762691 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-190724\"/\"kube-root-ca.crt\"" Nov 01 06:21:35.762853 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.762693 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-190724\"/\"default-dockercfg-52ts2\"" Nov 01 06:21:35.762853 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.762746 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-190724\"/\"openshift-service-ca.crt\"" Nov 01 06:21:35.762853 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.762751 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-190724\"/\"simulator-w4-190724-kserve-self-signed-certs\"" Nov 01 06:21:35.765849 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.765812 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx"] Nov 01 06:21:35.768447 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.768425 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:35.861313 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.861273 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6571db97-e767-4917-9543-5ebd4e6ff1e5-tls-certs\") pod \"simulator-w4-190724-kserve-5f758995c6-nz6xx\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:35.861516 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.861349 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-home\") pod \"simulator-w4-190724-kserve-5f758995c6-nz6xx\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:35.861516 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.861372 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-dshm\") pod \"simulator-w4-190724-kserve-5f758995c6-nz6xx\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:35.861629 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.861522 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-model-cache\") pod \"simulator-w4-190724-kserve-5f758995c6-nz6xx\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:35.862016 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.861990 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-home\") pod \"simulator-w4-190724-kserve-5f758995c6-nz6xx\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:35.862185 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.862134 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-tmp-dir\") pod \"simulator-w4-190724-kserve-5f758995c6-nz6xx\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:35.862276 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.862195 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mvpj\" (UniqueName: \"kubernetes.io/projected/6571db97-e767-4917-9543-5ebd4e6ff1e5-kube-api-access-4mvpj\") pod \"simulator-w4-190724-kserve-5f758995c6-nz6xx\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:35.862374 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.862016 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-model-cache\") pod \"simulator-w4-190724-kserve-5f758995c6-nz6xx\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:35.862477 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.862445 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-tmp-dir\") pod \"simulator-w4-190724-kserve-5f758995c6-nz6xx\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:35.864500 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.864471 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6571db97-e767-4917-9543-5ebd4e6ff1e5-tls-certs\") pod \"simulator-w4-190724-kserve-5f758995c6-nz6xx\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:35.864633 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.864613 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-dshm\") pod \"simulator-w4-190724-kserve-5f758995c6-nz6xx\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:35.919225 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.919149 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mvpj\" (UniqueName: \"kubernetes.io/projected/6571db97-e767-4917-9543-5ebd4e6ff1e5-kube-api-access-4mvpj\") pod \"simulator-w4-190724-kserve-5f758995c6-nz6xx\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:35.952892 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.952855 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-r5wn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 06:21:35.953381 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:35.952912 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" podUID="753c422d-48cc-40ee-8673-208ed65fd311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 06:21:36.002334 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:36.002032 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:36.404650 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:36.404604 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx"] Nov 01 06:21:36.405866 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:21:36.405814 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6571db97_e767_4917_9543_5ebd4e6ff1e5.slice/crio-46e283e7163d409e3ded396121a9e308b3bf38eb8331c16d6219ca885273385b WatchSource:0}: Error finding container 46e283e7163d409e3ded396121a9e308b3bf38eb8331c16d6219ca885273385b: Status 404 returned error can't find the container with id 46e283e7163d409e3ded396121a9e308b3bf38eb8331c16d6219ca885273385b Nov 01 06:21:36.533572 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:36.533538 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" event={"ID":"6571db97-e767-4917-9543-5ebd4e6ff1e5","Type":"ContainerStarted","Data":"2f0d8191dc1ce96678b6fd8843f9d9cff6769fe4e18d72512402eed1f099027c"} Nov 01 06:21:36.533572 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:36.533577 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" event={"ID":"6571db97-e767-4917-9543-5ebd4e6ff1e5","Type":"ContainerStarted","Data":"46e283e7163d409e3ded396121a9e308b3bf38eb8331c16d6219ca885273385b"} Nov 01 06:21:36.657773 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:36.657667 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" podStartSLOduration=1.6576520019999998 podStartE2EDuration="1.657652002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:21:36.655207419 +0000 UTC m=+706.419007277" watchObservedRunningTime="2026-11-01 06:21:36.657652002 +0000 UTC m=+706.421451899" Nov 01 06:21:36.799585 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:21:36.799546 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:21:36.952464 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:36.952387 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-r5wn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 06:21:36.952464 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:36.952445 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" podUID="753c422d-48cc-40ee-8673-208ed65fd311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 06:21:37.003126 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:37.003087 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:37.850342 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:37.850299 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:37.952611 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:37.952576 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-r5wn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 01 06:21:37.952788 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:37.952644 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" podUID="753c422d-48cc-40ee-8673-208ed65fd311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 01 06:21:38.045158 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:38.045119 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:39.953220 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:39.953187 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-r5wn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:21:39.953765 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:39.953737 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" podUID="753c422d-48cc-40ee-8673-208ed65fd311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:21:39.970582 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:39.970557 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:39.972074 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:39.972049 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:39.973930 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:39.973908 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:21:42.646974 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:42.646939 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:42.666775 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:42.666746 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:42.680562 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:42.680536 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:43.592232 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:43.592199 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 06:21:45.318452 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.318417 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59"] Nov 01 06:21:45.324685 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.324666 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.393009 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.392975 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.393164 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.393018 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.393164 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.393121 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.393279 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.393192 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.393343 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.393278 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.393343 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.393314 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.393446 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.393428 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.393502 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.393463 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.393502 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.393493 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvvxx\" (UniqueName: \"kubernetes.io/projected/7b22ecfe-990b-49a7-b565-f8b94668a5a1-kube-api-access-kvvxx\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.423498 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.423466 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59"] Nov 01 06:21:45.474867 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.474823 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2"] Nov 01 06:21:45.478898 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.478874 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:45.494230 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.494204 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8kdh\" (UniqueName: \"kubernetes.io/projected/27e2ec0f-708a-4ffb-9694-d1030859fc61-kube-api-access-k8kdh\") pod \"premium-w0-3ba2f3-kserve-5cd7896897-bxdj2\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:45.494384 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.494245 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.494384 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.494268 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-home\") pod \"premium-w0-3ba2f3-kserve-5cd7896897-bxdj2\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:45.494496 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.494395 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.494496 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.494428 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvvxx\" (UniqueName: \"kubernetes.io/projected/7b22ecfe-990b-49a7-b565-f8b94668a5a1-kube-api-access-kvvxx\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.494496 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.494452 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-model-cache\") pod \"premium-w0-3ba2f3-kserve-5cd7896897-bxdj2\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:45.494496 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.494478 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.494691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.494499 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.494691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.494527 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27e2ec0f-708a-4ffb-9694-d1030859fc61-tls-certs\") pod \"premium-w0-3ba2f3-kserve-5cd7896897-bxdj2\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:45.494691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.494588 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.494883 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.494775 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-dshm\") pod \"premium-w0-3ba2f3-kserve-5cd7896897-bxdj2\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:45.494952 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.494898 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-tmp-dir\") pod \"premium-w0-3ba2f3-kserve-5cd7896897-bxdj2\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:45.494952 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.494939 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.495054 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.494968 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.495054 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.494976 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.495054 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.495004 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.495054 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.495006 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.495285 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.495266 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.495352 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.495336 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.495916 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.495874 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.497701 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.497677 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.500438 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.500414 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.530892 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.530867 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3ba2f3\"/\"premium-w0-3ba2f3-kserve-self-signed-certs\"" Nov 01 06:21:45.532141 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.532115 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2"] Nov 01 06:21:45.551286 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.551259 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.551412 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.551348 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvvxx\" (UniqueName: \"kubernetes.io/projected/7b22ecfe-990b-49a7-b565-f8b94668a5a1-kube-api-access-kvvxx\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kvv59\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.573220 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.573145 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:45.595990 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.595962 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-dshm\") pod \"premium-w0-3ba2f3-kserve-5cd7896897-bxdj2\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:45.596160 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.596015 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-tmp-dir\") pod \"premium-w0-3ba2f3-kserve-5cd7896897-bxdj2\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:45.596160 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.596043 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8kdh\" (UniqueName: \"kubernetes.io/projected/27e2ec0f-708a-4ffb-9694-d1030859fc61-kube-api-access-k8kdh\") pod \"premium-w0-3ba2f3-kserve-5cd7896897-bxdj2\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:45.596160 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.596065 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-home\") pod \"premium-w0-3ba2f3-kserve-5cd7896897-bxdj2\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:45.596160 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.596088 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-model-cache\") pod \"premium-w0-3ba2f3-kserve-5cd7896897-bxdj2\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:45.596160 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.596119 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27e2ec0f-708a-4ffb-9694-d1030859fc61-tls-certs\") pod \"premium-w0-3ba2f3-kserve-5cd7896897-bxdj2\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:45.596534 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.596499 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-tmp-dir\") pod \"premium-w0-3ba2f3-kserve-5cd7896897-bxdj2\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:45.596534 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.596527 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-home\") pod \"premium-w0-3ba2f3-kserve-5cd7896897-bxdj2\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:45.596690 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.596572 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-model-cache\") pod \"premium-w0-3ba2f3-kserve-5cd7896897-bxdj2\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:45.598403 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.598378 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-dshm\") pod \"premium-w0-3ba2f3-kserve-5cd7896897-bxdj2\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:45.598749 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.598723 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27e2ec0f-708a-4ffb-9694-d1030859fc61-tls-certs\") pod \"premium-w0-3ba2f3-kserve-5cd7896897-bxdj2\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:45.638694 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.638665 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:45.652665 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.652634 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8kdh\" (UniqueName: \"kubernetes.io/projected/27e2ec0f-708a-4ffb-9694-d1030859fc61-kube-api-access-k8kdh\") pod \"premium-w0-3ba2f3-kserve-5cd7896897-bxdj2\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:45.792131 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:45.792098 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:46.003163 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:46.003131 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:46.015928 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:46.015904 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:46.065078 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:46.065054 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59"] Nov 01 06:21:46.065819 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:21:46.065796 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b22ecfe_990b_49a7_b565_f8b94668a5a1.slice/crio-17452e991af4c04557c4dd566373319a8c01b65e7b2973ed0f0b11255f933941 WatchSource:0}: Error finding container 17452e991af4c04557c4dd566373319a8c01b65e7b2973ed0f0b11255f933941: Status 404 returned error can't find the container with id 17452e991af4c04557c4dd566373319a8c01b65e7b2973ed0f0b11255f933941 Nov 01 06:21:46.068147 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:46.068118 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:21:46.068220 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:46.068176 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:21:46.068220 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:46.068209 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:21:46.170795 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:46.170761 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2"] Nov 01 06:21:46.171708 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:21:46.171676 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27e2ec0f_708a_4ffb_9694_d1030859fc61.slice/crio-282e5fa97115d2bc47d00be007a7c0bce6a90a90028733c30221161fe228c997 WatchSource:0}: Error finding container 282e5fa97115d2bc47d00be007a7c0bce6a90a90028733c30221161fe228c997: Status 404 returned error can't find the container with id 282e5fa97115d2bc47d00be007a7c0bce6a90a90028733c30221161fe228c997 Nov 01 06:21:46.603313 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:46.603269 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" event={"ID":"27e2ec0f-708a-4ffb-9694-d1030859fc61","Type":"ContainerStarted","Data":"5f72ad62f09e45fcdda20c9346d59ffb52ee2ec8d34be89cd1347588d075a679"} Nov 01 06:21:46.603793 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:46.603324 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" event={"ID":"27e2ec0f-708a-4ffb-9694-d1030859fc61","Type":"ContainerStarted","Data":"282e5fa97115d2bc47d00be007a7c0bce6a90a90028733c30221161fe228c997"} Nov 01 06:21:46.605007 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:46.604975 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" event={"ID":"7b22ecfe-990b-49a7-b565-f8b94668a5a1","Type":"ContainerStarted","Data":"76e8fb39e36691f5e94b922eec96b1ee3657a0222edee27d8d0cbea5a980d460"} Nov 01 06:21:46.605148 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:46.605060 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" event={"ID":"7b22ecfe-990b-49a7-b565-f8b94668a5a1","Type":"ContainerStarted","Data":"17452e991af4c04557c4dd566373319a8c01b65e7b2973ed0f0b11255f933941"} Nov 01 06:21:46.616933 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:46.616913 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 06:21:46.639575 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:46.639553 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:46.718096 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:46.718039 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" podStartSLOduration=1.718025594 podStartE2EDuration="1.718025594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:21:46.716751678 +0000 UTC m=+716.480551534" watchObservedRunningTime="2026-11-01 06:21:46.718025594 +0000 UTC m=+716.481825450" Nov 01 06:21:46.796022 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:46.795978 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:46.838033 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:46.837975 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" podStartSLOduration=1.837951911 podStartE2EDuration="1.837951911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:21:46.833661401 +0000 UTC m=+716.597461258" watchObservedRunningTime="2026-11-01 06:21:46.837951911 +0000 UTC m=+716.601751780" Nov 01 06:21:47.432637 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:47.432596 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:47.640924 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:47.640891 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kvv59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 06:21:47.641321 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:47.640947 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" podUID="7b22ecfe-990b-49a7-b565-f8b94668a5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 06:21:48.639794 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:48.639764 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kvv59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 06:21:48.639992 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:48.639807 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" podUID="7b22ecfe-990b-49a7-b565-f8b94668a5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 06:21:49.639578 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:49.639540 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kvv59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 06:21:49.640030 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:49.639613 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" podUID="7b22ecfe-990b-49a7-b565-f8b94668a5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 06:21:50.639126 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:50.639092 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kvv59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 06:21:50.639336 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:50.639145 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" podUID="7b22ecfe-990b-49a7-b565-f8b94668a5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 06:21:51.639850 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:51.639794 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kvv59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 06:21:51.640338 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:51.639881 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" podUID="7b22ecfe-990b-49a7-b565-f8b94668a5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 06:21:52.163869 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:52.163813 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:21:52.638982 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:52.638947 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kvv59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 06:21:52.639181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:52.638996 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" podUID="7b22ecfe-990b-49a7-b565-f8b94668a5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 06:21:54.392803 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:54.392769 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:54.395453 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:54.393307 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:54.395453 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:54.394651 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:21:55.793070 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:55.793044 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:55.805598 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:55.805576 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:56.676382 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:56.676353 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 06:21:56.929760 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:21:56.929675 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:01.629850 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:01.629804 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:06.102964 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:06.102927 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:06.532532 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:06.532440 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:06.689241 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:06.689210 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:06.779247 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:22:06.779215 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:22:07.137704 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:07.137672 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:07.414975 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:07.414879 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:11.935384 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:11.935351 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:12.089142 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:12.089099 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:15.039737 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:15.039701 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:15.230947 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:15.230912 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:15.589292 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:15.589259 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx"] Nov 01 06:22:15.594269 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:15.594241 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx" Nov 01 06:22:15.603943 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:15.603857 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7731ee91-483d-4396-8146-68e18afdd3c4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx\" (UID: \"7731ee91-483d-4396-8146-68e18afdd3c4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx" Nov 01 06:22:15.603943 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:15.603938 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jhjv\" (UniqueName: \"kubernetes.io/projected/7731ee91-483d-4396-8146-68e18afdd3c4-kube-api-access-7jhjv\") pod \"payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx\" (UID: \"7731ee91-483d-4396-8146-68e18afdd3c4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx" Nov 01 06:22:15.649177 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:15.649145 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx"] Nov 01 06:22:15.694659 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:15.694624 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ef9b55\"" Nov 01 06:22:15.694819 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:15.694728 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ef9b55-dockercfg-gjbzf\"" Nov 01 06:22:15.704600 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:15.704573 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7731ee91-483d-4396-8146-68e18afdd3c4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx\" (UID: \"7731ee91-483d-4396-8146-68e18afdd3c4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx" Nov 01 06:22:15.704740 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:15.704633 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jhjv\" (UniqueName: \"kubernetes.io/projected/7731ee91-483d-4396-8146-68e18afdd3c4-kube-api-access-7jhjv\") pod \"payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx\" (UID: \"7731ee91-483d-4396-8146-68e18afdd3c4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx" Nov 01 06:22:15.705328 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:15.705304 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7731ee91-483d-4396-8146-68e18afdd3c4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx\" (UID: \"7731ee91-483d-4396-8146-68e18afdd3c4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx" Nov 01 06:22:15.761137 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:15.761106 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jhjv\" (UniqueName: \"kubernetes.io/projected/7731ee91-483d-4396-8146-68e18afdd3c4-kube-api-access-7jhjv\") pod \"payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx\" (UID: \"7731ee91-483d-4396-8146-68e18afdd3c4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx" Nov 01 06:22:15.911491 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:15.911399 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx" Nov 01 06:22:16.287516 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:16.287489 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx"] Nov 01 06:22:16.295748 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:22:16.295720 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7731ee91_483d_4396_8146_68e18afdd3c4.slice/crio-fc7710402bc16a0661c97dbd9c051061060c4e9d8ded4d94365b45a9626a513a WatchSource:0}: Error finding container fc7710402bc16a0661c97dbd9c051061060c4e9d8ded4d94365b45a9626a513a: Status 404 returned error can't find the container with id fc7710402bc16a0661c97dbd9c051061060c4e9d8ded4d94365b45a9626a513a Nov 01 06:22:16.779526 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:16.779486 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx" event={"ID":"7731ee91-483d-4396-8146-68e18afdd3c4","Type":"ContainerStarted","Data":"53686ef8a8bd30cc543fd88da987803e64ea51365fff209b4de681a7e29d4139"} Nov 01 06:22:16.779709 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:16.779547 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx" event={"ID":"7731ee91-483d-4396-8146-68e18afdd3c4","Type":"ContainerStarted","Data":"fc7710402bc16a0661c97dbd9c051061060c4e9d8ded4d94365b45a9626a513a"} Nov 01 06:22:16.779709 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:16.779584 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx" Nov 01 06:22:16.894299 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:16.894249 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx" podStartSLOduration=1.894235026 podStartE2EDuration="1.894235026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:22:16.891308704 +0000 UTC m=+746.655108561" watchObservedRunningTime="2026-11-01 06:22:16.894235026 +0000 UTC m=+746.658034883" Nov 01 06:22:17.039891 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:17.039783 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:19.462141 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.462111 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl"] Nov 01 06:22:19.465737 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.465720 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:19.523707 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.523676 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl"] Nov 01 06:22:19.533588 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.533561 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-home\") pod \"premium-w4-190724-kserve-8599fcf989-dlzpl\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:19.533744 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.533594 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-model-cache\") pod \"premium-w4-190724-kserve-8599fcf989-dlzpl\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:19.533744 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.533612 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shxkf\" (UniqueName: \"kubernetes.io/projected/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-kube-api-access-shxkf\") pod \"premium-w4-190724-kserve-8599fcf989-dlzpl\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:19.533744 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.533672 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-tmp-dir\") pod \"premium-w4-190724-kserve-8599fcf989-dlzpl\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:19.533984 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.533755 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-dshm\") pod \"premium-w4-190724-kserve-8599fcf989-dlzpl\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:19.533984 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.533819 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-tls-certs\") pod \"premium-w4-190724-kserve-8599fcf989-dlzpl\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:19.566797 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.566775 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:19.567648 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.567628 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-190724\"/\"premium-w4-190724-kserve-self-signed-certs\"" Nov 01 06:22:19.634398 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.634364 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-home\") pod \"premium-w4-190724-kserve-8599fcf989-dlzpl\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:19.634398 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.634398 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-model-cache\") pod \"premium-w4-190724-kserve-8599fcf989-dlzpl\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:19.634660 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.634416 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shxkf\" (UniqueName: \"kubernetes.io/projected/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-kube-api-access-shxkf\") pod \"premium-w4-190724-kserve-8599fcf989-dlzpl\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:19.634792 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.634770 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-model-cache\") pod \"premium-w4-190724-kserve-8599fcf989-dlzpl\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:19.634877 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.634819 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-tmp-dir\") pod \"premium-w4-190724-kserve-8599fcf989-dlzpl\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:19.634959 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.634878 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-home\") pod \"premium-w4-190724-kserve-8599fcf989-dlzpl\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:19.634959 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.634890 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-dshm\") pod \"premium-w4-190724-kserve-8599fcf989-dlzpl\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:19.635065 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.634987 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-tls-certs\") pod \"premium-w4-190724-kserve-8599fcf989-dlzpl\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:19.635122 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.635076 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-tmp-dir\") pod \"premium-w4-190724-kserve-8599fcf989-dlzpl\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:19.637180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.637156 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-dshm\") pod \"premium-w4-190724-kserve-8599fcf989-dlzpl\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:19.637794 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.637776 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-tls-certs\") pod \"premium-w4-190724-kserve-8599fcf989-dlzpl\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:19.690765 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.690737 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shxkf\" (UniqueName: \"kubernetes.io/projected/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-kube-api-access-shxkf\") pod \"premium-w4-190724-kserve-8599fcf989-dlzpl\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:19.778284 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:19.778197 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:20.169716 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:20.169691 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl"] Nov 01 06:22:20.170017 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:22:20.169994 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5f0b488_c1f8_4a7e_9779_ebf4d94ad037.slice/crio-3cc6b350fef32e8810d25f7ab9834762c8c5d4edb6b6588238a068e20d38cf67 WatchSource:0}: Error finding container 3cc6b350fef32e8810d25f7ab9834762c8c5d4edb6b6588238a068e20d38cf67: Status 404 returned error can't find the container with id 3cc6b350fef32e8810d25f7ab9834762c8c5d4edb6b6588238a068e20d38cf67 Nov 01 06:22:20.806886 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:20.806851 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" event={"ID":"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037","Type":"ContainerStarted","Data":"0cd7d8c77186041bfa48a8177e7eed4e6d72e9f94da09402e031ed57d8d2b353"} Nov 01 06:22:20.806886 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:20.806892 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" event={"ID":"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037","Type":"ContainerStarted","Data":"3cc6b350fef32e8810d25f7ab9834762c8c5d4edb6b6588238a068e20d38cf67"} Nov 01 06:22:21.726355 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:21.726319 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:21.778626 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:21.778594 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:26.413447 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:26.413411 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:27.788160 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:27.788128 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx" Nov 01 06:22:27.901586 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:27.901524 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" podStartSLOduration=8.901508535 podStartE2EDuration="8.901508535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:22:20.920100467 +0000 UTC m=+750.683900325" watchObservedRunningTime="2026-11-01 06:22:27.901508535 +0000 UTC m=+757.665308392" Nov 01 06:22:29.779072 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:29.779040 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:29.793759 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:29.793729 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:29.874211 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:29.874181 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 06:22:31.123317 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:31.123277 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:35.763269 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:35.763238 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:35.922853 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:35.922797 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:36.070102 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:36.070011 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:36.781260 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:22:36.781224 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:22:38.243321 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:38.243287 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:40.748366 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:40.748330 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:42.122461 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:42.122428 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:45.327778 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:45.327738 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:50.119306 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:50.119266 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:50.514584 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:50.514508 4385 scope.go:122] "RemoveContainer" containerID="a05e1927e6e0cd643353c757d2aa077266e28c1e51d1b412f202619ca064be83" Nov 01 06:22:54.931397 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:54.931356 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:22:59.632130 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:22:59.632093 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:01.772257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:01.772207 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:02.392810 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:02.392772 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:02.579762 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:02.579720 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:04.335403 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:04.335364 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:06.280119 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:06.280083 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:06.489902 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:06.489865 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:06.779122 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:23:06.779089 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:23:09.396860 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:09.395472 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:09.550007 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:09.549970 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:09.700896 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:09.700773 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:10.038777 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:10.038700 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk"] Nov 01 06:23:10.042120 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:10.042097 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk" Nov 01 06:23:10.095164 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:10.094814 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c79abe\"" Nov 01 06:23:10.099779 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:10.099738 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk"] Nov 01 06:23:10.144623 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:10.144595 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c79abe-dockercfg-q6jdl\"" Nov 01 06:23:10.174096 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:10.174063 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv25s\" (UniqueName: \"kubernetes.io/projected/4be0c016-1f2d-4be4-a99a-ced55c192443-kube-api-access-jv25s\") pod \"payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk\" (UID: \"4be0c016-1f2d-4be4-a99a-ced55c192443\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk" Nov 01 06:23:10.174310 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:10.174143 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be0c016-1f2d-4be4-a99a-ced55c192443-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk\" (UID: \"4be0c016-1f2d-4be4-a99a-ced55c192443\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk" Nov 01 06:23:10.275437 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:10.275404 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv25s\" (UniqueName: \"kubernetes.io/projected/4be0c016-1f2d-4be4-a99a-ced55c192443-kube-api-access-jv25s\") pod \"payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk\" (UID: \"4be0c016-1f2d-4be4-a99a-ced55c192443\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk" Nov 01 06:23:10.275642 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:10.275473 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be0c016-1f2d-4be4-a99a-ced55c192443-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk\" (UID: \"4be0c016-1f2d-4be4-a99a-ced55c192443\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk" Nov 01 06:23:10.276227 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:10.276200 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be0c016-1f2d-4be4-a99a-ced55c192443-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk\" (UID: \"4be0c016-1f2d-4be4-a99a-ced55c192443\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk" Nov 01 06:23:10.331748 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:10.331717 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv25s\" (UniqueName: \"kubernetes.io/projected/4be0c016-1f2d-4be4-a99a-ced55c192443-kube-api-access-jv25s\") pod \"payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk\" (UID: \"4be0c016-1f2d-4be4-a99a-ced55c192443\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk" Nov 01 06:23:10.355860 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:10.355796 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk" Nov 01 06:23:10.727542 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:10.727513 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk"] Nov 01 06:23:10.728263 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:23:10.728238 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be0c016_1f2d_4be4_a99a_ced55c192443.slice/crio-e75fcba4a49147328191653599ac25b853cb0501a4aaf88f123c403756772a96 WatchSource:0}: Error finding container e75fcba4a49147328191653599ac25b853cb0501a4aaf88f123c403756772a96: Status 404 returned error can't find the container with id e75fcba4a49147328191653599ac25b853cb0501a4aaf88f123c403756772a96 Nov 01 06:23:11.137119 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:11.137084 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk" event={"ID":"4be0c016-1f2d-4be4-a99a-ced55c192443","Type":"ContainerStarted","Data":"a91912e5ae68959b8b969d85947253e66f93e46e5d50216156bab7536fd0def3"} Nov 01 06:23:11.137309 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:11.137144 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk" event={"ID":"4be0c016-1f2d-4be4-a99a-ced55c192443","Type":"ContainerStarted","Data":"e75fcba4a49147328191653599ac25b853cb0501a4aaf88f123c403756772a96"} Nov 01 06:23:11.137309 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:11.137172 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk" Nov 01 06:23:11.251907 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:11.251858 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk" podStartSLOduration=2.251827945 podStartE2EDuration="2.251827945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:23:11.251042939 +0000 UTC m=+801.014842807" watchObservedRunningTime="2026-11-01 06:23:11.251827945 +0000 UTC m=+801.015627800" Nov 01 06:23:12.569385 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:12.569351 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:14.586010 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:14.585978 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:18.152461 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.152422 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs"] Nov 01 06:23:18.156239 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.156224 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:18.212522 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.212493 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs"] Nov 01 06:23:18.256527 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.256490 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:18.258274 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.258257 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-85584b\"/\"openshift-service-ca.crt\"" Nov 01 06:23:18.258496 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.258476 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-85584b\"/\"kube-root-ca.crt\"" Nov 01 06:23:18.258575 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.258559 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-85584b\"/\"test-model-85584b-kserve-self-signed-certs\"" Nov 01 06:23:18.258622 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.258607 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-85584b\"/\"default-dockercfg-h8tr9\"" Nov 01 06:23:18.261034 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.261015 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-dshm\") pod \"test-model-85584b-kserve-646c97f487-4cjzs\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:18.261124 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.261045 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-tls-certs\") pod \"test-model-85584b-kserve-646c97f487-4cjzs\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:18.261124 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.261089 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-tmp-dir\") pod \"test-model-85584b-kserve-646c97f487-4cjzs\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:18.261235 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.261218 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-model-cache\") pod \"test-model-85584b-kserve-646c97f487-4cjzs\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:18.261278 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.261267 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn2dz\" (UniqueName: \"kubernetes.io/projected/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-kube-api-access-bn2dz\") pod \"test-model-85584b-kserve-646c97f487-4cjzs\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:18.261309 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.261300 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-home\") pod \"test-model-85584b-kserve-646c97f487-4cjzs\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:18.362614 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.362572 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn2dz\" (UniqueName: \"kubernetes.io/projected/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-kube-api-access-bn2dz\") pod \"test-model-85584b-kserve-646c97f487-4cjzs\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:18.362809 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.362645 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-home\") pod \"test-model-85584b-kserve-646c97f487-4cjzs\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:18.362809 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.362709 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-dshm\") pod \"test-model-85584b-kserve-646c97f487-4cjzs\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:18.362809 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.362735 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-tls-certs\") pod \"test-model-85584b-kserve-646c97f487-4cjzs\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:18.362809 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.362782 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-tmp-dir\") pod \"test-model-85584b-kserve-646c97f487-4cjzs\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:18.363105 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.362908 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-model-cache\") pod \"test-model-85584b-kserve-646c97f487-4cjzs\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:18.363385 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.363356 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-model-cache\") pod \"test-model-85584b-kserve-646c97f487-4cjzs\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:18.363520 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.363498 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-tmp-dir\") pod \"test-model-85584b-kserve-646c97f487-4cjzs\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:18.363589 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.363562 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-home\") pod \"test-model-85584b-kserve-646c97f487-4cjzs\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:18.365737 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.365708 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-dshm\") pod \"test-model-85584b-kserve-646c97f487-4cjzs\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:18.366103 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.366081 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-tls-certs\") pod \"test-model-85584b-kserve-646c97f487-4cjzs\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:18.419641 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.419566 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn2dz\" (UniqueName: \"kubernetes.io/projected/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-kube-api-access-bn2dz\") pod \"test-model-85584b-kserve-646c97f487-4cjzs\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:18.468471 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.468364 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:18.859653 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:18.859627 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs"] Nov 01 06:23:18.860703 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:23:18.860676 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6564d39_0b98_4ea2_9801_b31c8aa44ca3.slice/crio-34a063b1bcb4b4eb4da0e403b8ef64babcbecfbc5f705d5fee6464970da11d73 WatchSource:0}: Error finding container 34a063b1bcb4b4eb4da0e403b8ef64babcbecfbc5f705d5fee6464970da11d73: Status 404 returned error can't find the container with id 34a063b1bcb4b4eb4da0e403b8ef64babcbecfbc5f705d5fee6464970da11d73 Nov 01 06:23:19.187178 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:19.187098 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" event={"ID":"c6564d39-0b98-4ea2-9801-b31c8aa44ca3","Type":"ContainerStarted","Data":"bb0c25b7bd7c62cb1829a34af4c8403e8e53fa27031e0a3515476a65e395a6e8"} Nov 01 06:23:19.187178 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:19.187137 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" event={"ID":"c6564d39-0b98-4ea2-9801-b31c8aa44ca3","Type":"ContainerStarted","Data":"34a063b1bcb4b4eb4da0e403b8ef64babcbecfbc5f705d5fee6464970da11d73"} Nov 01 06:23:19.305220 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:19.305173 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" podStartSLOduration=1.305159927 podStartE2EDuration="1.305159927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:23:19.302299476 +0000 UTC m=+809.066099333" watchObservedRunningTime="2026-11-01 06:23:19.305159927 +0000 UTC m=+809.068959782" Nov 01 06:23:19.320765 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:19.320724 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:19.469390 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:19.469317 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:22.147088 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:22.147056 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk" Nov 01 06:23:24.016167 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:24.016129 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:28.468656 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:28.468619 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:28.481909 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:28.481878 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:28.727204 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:28.727121 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:29.259091 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:29.259062 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:23:33.438048 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:33.438002 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:36.778262 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:23:36.778227 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:23:38.059082 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:38.059040 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:38.261953 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:38.261916 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:38.408816 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:38.408782 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:38.572865 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:38.572810 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:42.910339 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:42.910302 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:46.631066 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:46.631030 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:47.633404 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:47.633363 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:52.461629 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:52.461585 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:55.557132 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.557098 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7"] Nov 01 06:23:55.561261 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.561238 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:23:55.612272 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.612241 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqljn\" (UniqueName: \"kubernetes.io/projected/d0029faa-cd51-43d8-8c9f-02399257ea72-kube-api-access-mqljn\") pod \"distinct-w4-190724-kserve-5f958f968d-zsnz7\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:23:55.612476 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.612316 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0029faa-cd51-43d8-8c9f-02399257ea72-tls-certs\") pod \"distinct-w4-190724-kserve-5f958f968d-zsnz7\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:23:55.612476 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.612408 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-model-cache\") pod \"distinct-w4-190724-kserve-5f958f968d-zsnz7\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:23:55.612476 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.612467 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-dshm\") pod \"distinct-w4-190724-kserve-5f958f968d-zsnz7\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:23:55.612772 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.612536 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-tmp-dir\") pod \"distinct-w4-190724-kserve-5f958f968d-zsnz7\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:23:55.612772 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.612651 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-home\") pod \"distinct-w4-190724-kserve-5f958f968d-zsnz7\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:23:55.617566 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.617543 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7"] Nov 01 06:23:55.662759 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.662731 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-190724\"/\"distinct-w4-190724-kserve-self-signed-certs\"" Nov 01 06:23:55.665662 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.665633 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:23:55.714161 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.714112 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-model-cache\") pod \"distinct-w4-190724-kserve-5f958f968d-zsnz7\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:23:55.714369 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.714184 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-dshm\") pod \"distinct-w4-190724-kserve-5f958f968d-zsnz7\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:23:55.714369 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.714220 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-tmp-dir\") pod \"distinct-w4-190724-kserve-5f958f968d-zsnz7\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:23:55.714522 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.714487 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-home\") pod \"distinct-w4-190724-kserve-5f958f968d-zsnz7\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:23:55.714600 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.714560 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqljn\" (UniqueName: \"kubernetes.io/projected/d0029faa-cd51-43d8-8c9f-02399257ea72-kube-api-access-mqljn\") pod \"distinct-w4-190724-kserve-5f958f968d-zsnz7\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:23:55.714665 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.714617 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-model-cache\") pod \"distinct-w4-190724-kserve-5f958f968d-zsnz7\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:23:55.714665 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.714625 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0029faa-cd51-43d8-8c9f-02399257ea72-tls-certs\") pod \"distinct-w4-190724-kserve-5f958f968d-zsnz7\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:23:55.715053 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.714997 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-home\") pod \"distinct-w4-190724-kserve-5f958f968d-zsnz7\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:23:55.715353 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.715174 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-tmp-dir\") pod \"distinct-w4-190724-kserve-5f958f968d-zsnz7\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:23:55.716900 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.716879 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-dshm\") pod \"distinct-w4-190724-kserve-5f958f968d-zsnz7\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:23:55.717632 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.717615 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0029faa-cd51-43d8-8c9f-02399257ea72-tls-certs\") pod \"distinct-w4-190724-kserve-5f958f968d-zsnz7\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:23:55.770948 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.770922 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqljn\" (UniqueName: \"kubernetes.io/projected/d0029faa-cd51-43d8-8c9f-02399257ea72-kube-api-access-mqljn\") pod \"distinct-w4-190724-kserve-5f958f968d-zsnz7\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:23:55.905810 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:55.905772 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:23:56.321373 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:56.321342 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7"] Nov 01 06:23:56.322260 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:23:56.322231 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0029faa_cd51_43d8_8c9f_02399257ea72.slice/crio-7439f0b030aaa7e719a15cec21ffe782d6e4eaa31eb423086efa46e8add43d4d WatchSource:0}: Error finding container 7439f0b030aaa7e719a15cec21ffe782d6e4eaa31eb423086efa46e8add43d4d: Status 404 returned error can't find the container with id 7439f0b030aaa7e719a15cec21ffe782d6e4eaa31eb423086efa46e8add43d4d Nov 01 06:23:56.410602 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:56.410565 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" event={"ID":"d0029faa-cd51-43d8-8c9f-02399257ea72","Type":"ContainerStarted","Data":"566d6efd0d2778af3c0370d57b15d242426064a7d8cb8687684e103785500c4b"} Nov 01 06:23:56.410792 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:56.410634 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" event={"ID":"d0029faa-cd51-43d8-8c9f-02399257ea72","Type":"ContainerStarted","Data":"7439f0b030aaa7e719a15cec21ffe782d6e4eaa31eb423086efa46e8add43d4d"} Nov 01 06:23:56.526955 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:56.526906 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" podStartSLOduration=1.5268909910000001 podStartE2EDuration="1.526890991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:23:56.52355919 +0000 UTC m=+846.287359050" watchObservedRunningTime="2026-11-01 06:23:56.526890991 +0000 UTC m=+846.290690848" Nov 01 06:23:56.907027 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:56.906894 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:23:57.150817 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:23:57.150788 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:01.958037 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:01.957932 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:05.906745 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:05.906707 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:24:05.924520 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:05.924492 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:24:06.484627 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:06.484602 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 06:24:06.730581 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:06.730545 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:06.783862 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:24:06.783753 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:24:11.436723 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:11.436687 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:14.299692 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:14.299656 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn"] Nov 01 06:24:14.303318 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:14.303289 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn" Nov 01 06:24:14.359252 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:14.359221 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn"] Nov 01 06:24:14.404643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:14.404613 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:24:14.404802 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:14.404658 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\"" Nov 01 06:24:14.409431 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:14.409407 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c96d1a13-d35d-4144-b5cb-090ef28d84f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51a6dfc6-fbngn\" (UID: \"c96d1a13-d35d-4144-b5cb-090ef28d84f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn" Nov 01 06:24:14.409567 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:14.409546 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrpbl\" (UniqueName: \"kubernetes.io/projected/c96d1a13-d35d-4144-b5cb-090ef28d84f1-kube-api-access-mrpbl\") pod \"maas-api-revoke-sub-51a6dfc6-fbngn\" (UID: \"c96d1a13-d35d-4144-b5cb-090ef28d84f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn" Nov 01 06:24:14.409613 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:14.409604 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c96d1a13-d35d-4144-b5cb-090ef28d84f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51a6dfc6-fbngn\" (UID: \"c96d1a13-d35d-4144-b5cb-090ef28d84f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn" Nov 01 06:24:14.511089 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:14.511059 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrpbl\" (UniqueName: \"kubernetes.io/projected/c96d1a13-d35d-4144-b5cb-090ef28d84f1-kube-api-access-mrpbl\") pod \"maas-api-revoke-sub-51a6dfc6-fbngn\" (UID: \"c96d1a13-d35d-4144-b5cb-090ef28d84f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn" Nov 01 06:24:14.511248 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:14.511112 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c96d1a13-d35d-4144-b5cb-090ef28d84f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51a6dfc6-fbngn\" (UID: \"c96d1a13-d35d-4144-b5cb-090ef28d84f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn" Nov 01 06:24:14.511248 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:14.511188 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c96d1a13-d35d-4144-b5cb-090ef28d84f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51a6dfc6-fbngn\" (UID: \"c96d1a13-d35d-4144-b5cb-090ef28d84f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn" Nov 01 06:24:14.511958 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:14.511929 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c96d1a13-d35d-4144-b5cb-090ef28d84f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51a6dfc6-fbngn\" (UID: \"c96d1a13-d35d-4144-b5cb-090ef28d84f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn" Nov 01 06:24:14.513672 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:14.513650 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c96d1a13-d35d-4144-b5cb-090ef28d84f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51a6dfc6-fbngn\" (UID: \"c96d1a13-d35d-4144-b5cb-090ef28d84f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn" Nov 01 06:24:14.567220 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:14.567165 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrpbl\" (UniqueName: \"kubernetes.io/projected/c96d1a13-d35d-4144-b5cb-090ef28d84f1-kube-api-access-mrpbl\") pod \"maas-api-revoke-sub-51a6dfc6-fbngn\" (UID: \"c96d1a13-d35d-4144-b5cb-090ef28d84f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn" Nov 01 06:24:14.616571 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:14.616540 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn" Nov 01 06:24:14.994200 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:14.994168 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn"] Nov 01 06:24:14.998294 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:24:14.998261 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc96d1a13_d35d_4144_b5cb_090ef28d84f1.slice/crio-fdadd9147e7a7bbe1316bf8b5382c8e41d9580c7b55ee1137d795700a8790a34 WatchSource:0}: Error finding container fdadd9147e7a7bbe1316bf8b5382c8e41d9580c7b55ee1137d795700a8790a34: Status 404 returned error can't find the container with id fdadd9147e7a7bbe1316bf8b5382c8e41d9580c7b55ee1137d795700a8790a34 Nov 01 06:24:15.531705 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:15.531674 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn" event={"ID":"c96d1a13-d35d-4144-b5cb-090ef28d84f1","Type":"ContainerStarted","Data":"fdadd9147e7a7bbe1316bf8b5382c8e41d9580c7b55ee1137d795700a8790a34"} Nov 01 06:24:16.143167 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:16.143130 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:17.433720 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:17.433647 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:18.674948 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:18.674905 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:19.232644 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:19.232548 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:19.470312 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:19.470273 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:19.558951 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:19.558859 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn" event={"ID":"c96d1a13-d35d-4144-b5cb-090ef28d84f1","Type":"ContainerStarted","Data":"93883262fd44a16701a278f12a65835b61840fb5612fa5109bdd96d4e6633e38"} Nov 01 06:24:19.671948 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:19.671808 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn" podStartSLOduration=1.997280608 podStartE2EDuration="5.671790888s" totalImagesPullingTime="3.67451028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:24:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:24:19.67110021 +0000 UTC m=+869.434900067" watchObservedRunningTime="2026-11-01 06:24:19.671790888 +0000 UTC m=+869.435590746" Nov 01 06:24:20.566147 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:20.566118 4385 generic.go:356] "Generic (PLEG): container finished" podID="c96d1a13-d35d-4144-b5cb-090ef28d84f1" containerID="93883262fd44a16701a278f12a65835b61840fb5612fa5109bdd96d4e6633e38" exitCode=0 Nov 01 06:24:20.566549 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:20.566196 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn" event={"ID":"c96d1a13-d35d-4144-b5cb-090ef28d84f1","Type":"ContainerDied","Data":"93883262fd44a16701a278f12a65835b61840fb5612fa5109bdd96d4e6633e38"} Nov 01 06:24:21.139634 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:21.139599 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:21.716218 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:21.716197 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn" Nov 01 06:24:21.790919 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:21.790893 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c96d1a13-d35d-4144-b5cb-090ef28d84f1-kube-api-access-mrpbl\" (UniqueName: \"kubernetes.io/projected/c96d1a13-d35d-4144-b5cb-090ef28d84f1-kube-api-access-mrpbl\") pod \"c96d1a13-d35d-4144-b5cb-090ef28d84f1\" (UID: \"c96d1a13-d35d-4144-b5cb-090ef28d84f1\") " Nov 01 06:24:21.791083 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:21.790961 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c96d1a13-d35d-4144-b5cb-090ef28d84f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c96d1a13-d35d-4144-b5cb-090ef28d84f1-maas-api-xks-ca\") pod \"c96d1a13-d35d-4144-b5cb-090ef28d84f1\" (UID: \"c96d1a13-d35d-4144-b5cb-090ef28d84f1\") " Nov 01 06:24:21.791083 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:21.790986 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c96d1a13-d35d-4144-b5cb-090ef28d84f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c96d1a13-d35d-4144-b5cb-090ef28d84f1-maas-api-openshift-ca\") pod \"c96d1a13-d35d-4144-b5cb-090ef28d84f1\" (UID: \"c96d1a13-d35d-4144-b5cb-090ef28d84f1\") " Nov 01 06:24:21.791446 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:21.791415 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c96d1a13-d35d-4144-b5cb-090ef28d84f1-maas-api-openshift-ca" pod "c96d1a13-d35d-4144-b5cb-090ef28d84f1" (UID: "c96d1a13-d35d-4144-b5cb-090ef28d84f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:24:21.793224 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:21.793199 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c96d1a13-d35d-4144-b5cb-090ef28d84f1-kube-api-access-mrpbl" pod "c96d1a13-d35d-4144-b5cb-090ef28d84f1" (UID: "c96d1a13-d35d-4144-b5cb-090ef28d84f1"). InnerVolumeSpecName "kube-api-access-mrpbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:24:21.793312 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:21.793254 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c96d1a13-d35d-4144-b5cb-090ef28d84f1-maas-api-xks-ca" pod "c96d1a13-d35d-4144-b5cb-090ef28d84f1" (UID: "c96d1a13-d35d-4144-b5cb-090ef28d84f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:24:21.891764 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:21.891736 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c96d1a13-d35d-4144-b5cb-090ef28d84f1-maas-api-xks-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:21.891764 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:21.891762 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c96d1a13-d35d-4144-b5cb-090ef28d84f1-maas-api-openshift-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:21.891987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:21.891774 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrpbl\" (UniqueName: \"kubernetes.io/projected/c96d1a13-d35d-4144-b5cb-090ef28d84f1-kube-api-access-mrpbl\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:24:22.579628 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:22.579603 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn" Nov 01 06:24:22.579868 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:22.579600 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn" event={"ID":"c96d1a13-d35d-4144-b5cb-090ef28d84f1","Type":"ContainerDied","Data":"fdadd9147e7a7bbe1316bf8b5382c8e41d9580c7b55ee1137d795700a8790a34"} Nov 01 06:24:22.579868 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:22.579731 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdadd9147e7a7bbe1316bf8b5382c8e41d9580c7b55ee1137d795700a8790a34" Nov 01 06:24:23.797637 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:23.797600 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:23.962075 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:23.962040 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:24.120755 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:24.120701 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:26.122368 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.122324 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:26.645263 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.645226 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c"] Nov 01 06:24:26.645731 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.645713 4385 cpu_manager.go:414] "containerMap: removing container" podUID="c96d1a13-d35d-4144-b5cb-090ef28d84f1" containerName="revoke-keys" Nov 01 06:24:26.645882 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.645735 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c96d1a13-d35d-4144-b5cb-090ef28d84f1" containerName="revoke-keys" Nov 01 06:24:26.645958 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.645894 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="c96d1a13-d35d-4144-b5cb-090ef28d84f1" containerName="revoke-keys" Nov 01 06:24:26.649547 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.649525 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.701505 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.701477 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-190724-openshift-default-dockercfg-vr7pw\"" Nov 01 06:24:26.728719 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.728691 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.728875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.728733 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.728875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.728823 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.728948 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.728910 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.728983 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.728945 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.729023 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.728981 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.729023 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.728999 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.729156 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.729136 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.729213 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.729179 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96mgf\" (UniqueName: \"kubernetes.io/projected/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-kube-api-access-96mgf\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.756344 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.756318 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c"] Nov 01 06:24:26.830640 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.830596 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.830640 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.830661 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.830968 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.830688 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.830968 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.830719 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.830968 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.830738 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.830968 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.830781 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.830968 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.830805 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.830968 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.830916 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.830968 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.830970 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96mgf\" (UniqueName: \"kubernetes.io/projected/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-kube-api-access-96mgf\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.831335 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.831080 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.831335 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.831287 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.831335 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.831313 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.831485 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.831414 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.831485 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.831475 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.833291 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.833272 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.833379 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.833362 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.887242 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.887210 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.887477 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.887453 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96mgf\" (UniqueName: \"kubernetes.io/projected/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-kube-api-access-96mgf\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:26.962195 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:26.962123 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:27.065018 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:27.064980 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:27.393063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:27.393038 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c"] Nov 01 06:24:27.394049 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:24:27.394023 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010ccc17_ec4d_4ec3_b3c9_0704c9982fd4.slice/crio-05745a1b63954b13851bc9f278764985f5054bae315cada50b6f8933c0ad01fa WatchSource:0}: Error finding container 05745a1b63954b13851bc9f278764985f5054bae315cada50b6f8933c0ad01fa: Status 404 returned error can't find the container with id 05745a1b63954b13851bc9f278764985f5054bae315cada50b6f8933c0ad01fa Nov 01 06:24:27.396443 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:27.396411 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:24:27.396549 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:27.396489 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:24:27.396549 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:27.396531 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:24:27.627008 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:27.626962 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" event={"ID":"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4","Type":"ContainerStarted","Data":"e9d8e25ac8cac9c328708418d97f4506cf28eb4e03e6bb999ddfaafa832a4443"} Nov 01 06:24:27.627224 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:27.627045 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" event={"ID":"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4","Type":"ContainerStarted","Data":"05745a1b63954b13851bc9f278764985f5054bae315cada50b6f8933c0ad01fa"} Nov 01 06:24:27.746731 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:27.746640 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" podStartSLOduration=1.746625179 podStartE2EDuration="1.746625179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:24:27.743683401 +0000 UTC m=+877.507483269" watchObservedRunningTime="2026-11-01 06:24:27.746625179 +0000 UTC m=+877.510425036" Nov 01 06:24:27.962539 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:27.962503 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:28.614265 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:28.614234 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:28.964078 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:28.964000 4385 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 06:24:28.964078 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:28.964051 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" podUID="010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 06:24:29.963199 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:29.963163 4385 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 06:24:29.963603 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:29.963231 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" podUID="010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 06:24:30.962777 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:30.962743 4385 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 06:24:30.962983 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:30.962796 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" podUID="010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 06:24:31.127036 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:31.126991 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:31.962513 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:31.962477 4385 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 06:24:31.962734 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:31.962534 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" podUID="010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 06:24:32.963174 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:32.963140 4385 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 06:24:32.963587 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:32.963192 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" podUID="010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 06:24:33.962717 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:33.962684 4385 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 01 06:24:33.962910 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:33.962736 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" podUID="010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 01 06:24:34.967448 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:34.967417 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:34.967888 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:34.967683 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:34.968379 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:34.968359 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:24:35.830048 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:35.830006 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:36.779678 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:24:36.779641 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:24:37.708040 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:37.708005 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:38.907608 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:38.907572 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:40.494587 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:40.494548 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:45.117623 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:45.117591 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:49.849014 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:49.848977 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:54.533346 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:54.533304 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:59.229492 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:59.229447 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:24:59.425539 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:24:59.425504 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:04.046762 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:04.046726 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:06.780162 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:25:06.780119 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:25:08.904555 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:08.904517 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:11.996589 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:11.996556 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55"] Nov 01 06:25:12.000033 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.000017 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.066456 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.066403 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21387363-d464-4483-9f27-d9fcc7e72804-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.066666 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.066467 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.066666 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.066497 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdh62\" (UniqueName: \"kubernetes.io/projected/21387363-d464-4483-9f27-d9fcc7e72804-kube-api-access-mdh62\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.066666 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.066594 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.066666 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.066631 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.066990 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.066688 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.066990 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.066739 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21387363-d464-4483-9f27-d9fcc7e72804-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.066990 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.066788 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.066990 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.066812 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21387363-d464-4483-9f27-d9fcc7e72804-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.102227 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.102198 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55"] Nov 01 06:25:12.167680 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.167652 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.167680 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.167684 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdh62\" (UniqueName: \"kubernetes.io/projected/21387363-d464-4483-9f27-d9fcc7e72804-kube-api-access-mdh62\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.167976 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.167729 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.167976 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.167756 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.167976 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.167945 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.168140 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.168026 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21387363-d464-4483-9f27-d9fcc7e72804-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.168140 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.168083 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.168140 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.168111 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21387363-d464-4483-9f27-d9fcc7e72804-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.168311 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.168189 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21387363-d464-4483-9f27-d9fcc7e72804-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.168311 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.168208 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.168423 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.168342 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.168423 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.168394 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.168774 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.168746 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.168925 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.168859 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21387363-d464-4483-9f27-d9fcc7e72804-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.171621 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.171576 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21387363-d464-4483-9f27-d9fcc7e72804-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.172184 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.172164 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.225912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.225826 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21387363-d464-4483-9f27-d9fcc7e72804-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.226070 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.225923 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdh62\" (UniqueName: \"kubernetes.io/projected/21387363-d464-4483-9f27-d9fcc7e72804-kube-api-access-mdh62\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.312969 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.312889 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:12.741616 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:25:12.741584 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21387363_d464_4483_9f27_d9fcc7e72804.slice/crio-19e3656e30c943b66d1b512c3d60440aa5d9c641185806392811a4f4617a8784 WatchSource:0}: Error finding container 19e3656e30c943b66d1b512c3d60440aa5d9c641185806392811a4f4617a8784: Status 404 returned error can't find the container with id 19e3656e30c943b66d1b512c3d60440aa5d9c641185806392811a4f4617a8784 Nov 01 06:25:12.741945 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.741918 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55"] Nov 01 06:25:12.744311 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.744277 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:25:12.744400 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.744383 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:25:12.744439 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.744429 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:25:12.761321 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.761301 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr"] Nov 01 06:25:12.766019 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.765993 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr" Nov 01 06:25:12.818910 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.818882 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:25:12.818910 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.818882 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\"" Nov 01 06:25:12.820629 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.820606 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr"] Nov 01 06:25:12.877898 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.877855 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ba053c1-ca6c-4dbf-b96d-091354a3c245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a80c5c9-g72tr\" (UID: \"0ba053c1-ca6c-4dbf-b96d-091354a3c245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr" Nov 01 06:25:12.878089 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.877980 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94szl\" (UniqueName: \"kubernetes.io/projected/0ba053c1-ca6c-4dbf-b96d-091354a3c245-kube-api-access-94szl\") pod \"maas-api-revoke-sub-0a80c5c9-g72tr\" (UID: \"0ba053c1-ca6c-4dbf-b96d-091354a3c245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr" Nov 01 06:25:12.878089 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.878059 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ba053c1-ca6c-4dbf-b96d-091354a3c245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a80c5c9-g72tr\" (UID: \"0ba053c1-ca6c-4dbf-b96d-091354a3c245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr" Nov 01 06:25:12.923024 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.922992 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" event={"ID":"21387363-d464-4483-9f27-d9fcc7e72804","Type":"ContainerStarted","Data":"cae687cef88a6625c5d098f7b61e787eb6691c8f46aa484a5a0329b6464347b3"} Nov 01 06:25:12.923218 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.923052 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" event={"ID":"21387363-d464-4483-9f27-d9fcc7e72804","Type":"ContainerStarted","Data":"19e3656e30c943b66d1b512c3d60440aa5d9c641185806392811a4f4617a8784"} Nov 01 06:25:12.979116 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.979087 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94szl\" (UniqueName: \"kubernetes.io/projected/0ba053c1-ca6c-4dbf-b96d-091354a3c245-kube-api-access-94szl\") pod \"maas-api-revoke-sub-0a80c5c9-g72tr\" (UID: \"0ba053c1-ca6c-4dbf-b96d-091354a3c245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr" Nov 01 06:25:12.979304 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.979128 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ba053c1-ca6c-4dbf-b96d-091354a3c245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a80c5c9-g72tr\" (UID: \"0ba053c1-ca6c-4dbf-b96d-091354a3c245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr" Nov 01 06:25:12.979304 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.979258 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ba053c1-ca6c-4dbf-b96d-091354a3c245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a80c5c9-g72tr\" (UID: \"0ba053c1-ca6c-4dbf-b96d-091354a3c245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr" Nov 01 06:25:12.979929 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.979905 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ba053c1-ca6c-4dbf-b96d-091354a3c245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a80c5c9-g72tr\" (UID: \"0ba053c1-ca6c-4dbf-b96d-091354a3c245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr" Nov 01 06:25:12.981672 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:12.981648 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ba053c1-ca6c-4dbf-b96d-091354a3c245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a80c5c9-g72tr\" (UID: \"0ba053c1-ca6c-4dbf-b96d-091354a3c245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr" Nov 01 06:25:13.035749 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:13.035660 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94szl\" (UniqueName: \"kubernetes.io/projected/0ba053c1-ca6c-4dbf-b96d-091354a3c245-kube-api-access-94szl\") pod \"maas-api-revoke-sub-0a80c5c9-g72tr\" (UID: \"0ba053c1-ca6c-4dbf-b96d-091354a3c245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr" Nov 01 06:25:13.040026 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:13.039979 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" podStartSLOduration=2.039966147 podStartE2EDuration="2.039966147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:13.037570851 +0000 UTC m=+922.801370707" watchObservedRunningTime="2026-11-01 06:25:13.039966147 +0000 UTC m=+922.803766064" Nov 01 06:25:13.090573 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:13.090543 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr" Nov 01 06:25:13.314126 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:13.314033 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:13.316369 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:13.316344 4385 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 06:25:13.316486 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:13.316400 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" podUID="21387363-d464-4483-9f27-d9fcc7e72804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 06:25:13.469695 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:13.469668 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr"] Nov 01 06:25:13.479538 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:25:13.479442 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba053c1_ca6c_4dbf_b96d_091354a3c245.slice/crio-f79fbc03ba1118f7b0357690fd732c33c1f57318ce505a7aeac0e29f642488d9 WatchSource:0}: Error finding container f79fbc03ba1118f7b0357690fd732c33c1f57318ce505a7aeac0e29f642488d9: Status 404 returned error can't find the container with id f79fbc03ba1118f7b0357690fd732c33c1f57318ce505a7aeac0e29f642488d9 Nov 01 06:25:13.631745 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:13.631713 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:13.934043 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:13.933998 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr" event={"ID":"0ba053c1-ca6c-4dbf-b96d-091354a3c245","Type":"ContainerStarted","Data":"78b855e648e3ea909965e0e28e78f9318429f6feb1baab8b02406d98ab5e19fa"} Nov 01 06:25:13.934212 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:13.934113 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr" event={"ID":"0ba053c1-ca6c-4dbf-b96d-091354a3c245","Type":"ContainerStarted","Data":"f79fbc03ba1118f7b0357690fd732c33c1f57318ce505a7aeac0e29f642488d9"} Nov 01 06:25:14.047972 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:14.047919 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr" podStartSLOduration=2.047906382 podStartE2EDuration="2.047906382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:14.046792524 +0000 UTC m=+923.810592382" watchObservedRunningTime="2026-11-01 06:25:14.047906382 +0000 UTC m=+923.811706239" Nov 01 06:25:14.313539 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:14.313431 4385 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 06:25:14.313539 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:14.313511 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" podUID="21387363-d464-4483-9f27-d9fcc7e72804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 06:25:14.942640 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:14.942605 4385 generic.go:356] "Generic (PLEG): container finished" podID="0ba053c1-ca6c-4dbf-b96d-091354a3c245" containerID="78b855e648e3ea909965e0e28e78f9318429f6feb1baab8b02406d98ab5e19fa" exitCode=0 Nov 01 06:25:14.942820 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:14.942685 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr" event={"ID":"0ba053c1-ca6c-4dbf-b96d-091354a3c245","Type":"ContainerDied","Data":"78b855e648e3ea909965e0e28e78f9318429f6feb1baab8b02406d98ab5e19fa"} Nov 01 06:25:15.313545 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:15.313512 4385 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 06:25:15.313979 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:15.313572 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" podUID="21387363-d464-4483-9f27-d9fcc7e72804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 06:25:16.157367 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:16.157342 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr" Nov 01 06:25:16.313458 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:16.313355 4385 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 06:25:16.313458 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:16.313407 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" podUID="21387363-d464-4483-9f27-d9fcc7e72804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 06:25:16.333004 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:16.332968 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ba053c1-ca6c-4dbf-b96d-091354a3c245-kube-api-access-94szl\" (UniqueName: \"kubernetes.io/projected/0ba053c1-ca6c-4dbf-b96d-091354a3c245-kube-api-access-94szl\") pod \"0ba053c1-ca6c-4dbf-b96d-091354a3c245\" (UID: \"0ba053c1-ca6c-4dbf-b96d-091354a3c245\") " Nov 01 06:25:16.333239 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:16.333053 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ba053c1-ca6c-4dbf-b96d-091354a3c245-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ba053c1-ca6c-4dbf-b96d-091354a3c245-maas-api-xks-ca\") pod \"0ba053c1-ca6c-4dbf-b96d-091354a3c245\" (UID: \"0ba053c1-ca6c-4dbf-b96d-091354a3c245\") " Nov 01 06:25:16.333239 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:16.333194 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ba053c1-ca6c-4dbf-b96d-091354a3c245-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ba053c1-ca6c-4dbf-b96d-091354a3c245-maas-api-openshift-ca\") pod \"0ba053c1-ca6c-4dbf-b96d-091354a3c245\" (UID: \"0ba053c1-ca6c-4dbf-b96d-091354a3c245\") " Nov 01 06:25:16.333556 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:16.333529 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ba053c1-ca6c-4dbf-b96d-091354a3c245-maas-api-openshift-ca" pod "0ba053c1-ca6c-4dbf-b96d-091354a3c245" (UID: "0ba053c1-ca6c-4dbf-b96d-091354a3c245"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:25:16.335953 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:16.335923 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ba053c1-ca6c-4dbf-b96d-091354a3c245-maas-api-xks-ca" pod "0ba053c1-ca6c-4dbf-b96d-091354a3c245" (UID: "0ba053c1-ca6c-4dbf-b96d-091354a3c245"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:25:16.336058 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:16.335960 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba053c1-ca6c-4dbf-b96d-091354a3c245-kube-api-access-94szl" pod "0ba053c1-ca6c-4dbf-b96d-091354a3c245" (UID: "0ba053c1-ca6c-4dbf-b96d-091354a3c245"). InnerVolumeSpecName "kube-api-access-94szl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:25:16.435111 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:16.435069 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ba053c1-ca6c-4dbf-b96d-091354a3c245-maas-api-openshift-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:16.435111 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:16.435098 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94szl\" (UniqueName: \"kubernetes.io/projected/0ba053c1-ca6c-4dbf-b96d-091354a3c245-kube-api-access-94szl\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:16.435111 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:16.435108 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ba053c1-ca6c-4dbf-b96d-091354a3c245-maas-api-xks-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:16.955758 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:16.955733 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr" Nov 01 06:25:16.955960 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:16.955773 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr" event={"ID":"0ba053c1-ca6c-4dbf-b96d-091354a3c245","Type":"ContainerDied","Data":"f79fbc03ba1118f7b0357690fd732c33c1f57318ce505a7aeac0e29f642488d9"} Nov 01 06:25:16.955960 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:16.955828 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f79fbc03ba1118f7b0357690fd732c33c1f57318ce505a7aeac0e29f642488d9" Nov 01 06:25:17.313824 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:17.313735 4385 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 06:25:17.313824 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:17.313793 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" podUID="21387363-d464-4483-9f27-d9fcc7e72804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 06:25:18.313353 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:18.313312 4385 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 06:25:18.313518 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:18.313368 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" podUID="21387363-d464-4483-9f27-d9fcc7e72804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 06:25:18.360901 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:18.360863 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:19.314195 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:19.314155 4385 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 01 06:25:19.314387 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:19.314213 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" podUID="21387363-d464-4483-9f27-d9fcc7e72804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 01 06:25:20.319372 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:20.319340 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:20.319883 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:20.319709 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:20.320506 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:20.320490 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:25:23.151531 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:23.151496 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:23.511459 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:23.511379 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b"] Nov 01 06:25:23.511827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:23.511816 4385 cpu_manager.go:414] "containerMap: removing container" podUID="0ba053c1-ca6c-4dbf-b96d-091354a3c245" containerName="revoke-keys" Nov 01 06:25:23.511894 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:23.511842 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ba053c1-ca6c-4dbf-b96d-091354a3c245" containerName="revoke-keys" Nov 01 06:25:23.511938 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:23.511929 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ba053c1-ca6c-4dbf-b96d-091354a3c245" containerName="revoke-keys" Nov 01 06:25:23.515462 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:23.515445 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b" Nov 01 06:25:23.570980 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:23.570944 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b"] Nov 01 06:25:23.615964 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:23.615924 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:25:23.616139 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:23.615992 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\"" Nov 01 06:25:23.622073 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:23.622047 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/957bb8ef-8ebd-4f16-8c28-254b741c5b5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b77cc454-l867b\" (UID: \"957bb8ef-8ebd-4f16-8c28-254b741c5b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b" Nov 01 06:25:23.622199 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:23.622081 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nf49\" (UniqueName: \"kubernetes.io/projected/957bb8ef-8ebd-4f16-8c28-254b741c5b5e-kube-api-access-8nf49\") pod \"maas-api-revoke-sub-b77cc454-l867b\" (UID: \"957bb8ef-8ebd-4f16-8c28-254b741c5b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b" Nov 01 06:25:23.622199 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:23.622183 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/957bb8ef-8ebd-4f16-8c28-254b741c5b5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b77cc454-l867b\" (UID: \"957bb8ef-8ebd-4f16-8c28-254b741c5b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b" Nov 01 06:25:23.723288 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:23.723248 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/957bb8ef-8ebd-4f16-8c28-254b741c5b5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b77cc454-l867b\" (UID: \"957bb8ef-8ebd-4f16-8c28-254b741c5b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b" Nov 01 06:25:23.723288 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:23.723289 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nf49\" (UniqueName: \"kubernetes.io/projected/957bb8ef-8ebd-4f16-8c28-254b741c5b5e-kube-api-access-8nf49\") pod \"maas-api-revoke-sub-b77cc454-l867b\" (UID: \"957bb8ef-8ebd-4f16-8c28-254b741c5b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b" Nov 01 06:25:23.723513 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:23.723324 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/957bb8ef-8ebd-4f16-8c28-254b741c5b5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b77cc454-l867b\" (UID: \"957bb8ef-8ebd-4f16-8c28-254b741c5b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b" Nov 01 06:25:23.723941 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:23.723892 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/957bb8ef-8ebd-4f16-8c28-254b741c5b5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b77cc454-l867b\" (UID: \"957bb8ef-8ebd-4f16-8c28-254b741c5b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b" Nov 01 06:25:23.725984 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:23.725960 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/957bb8ef-8ebd-4f16-8c28-254b741c5b5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b77cc454-l867b\" (UID: \"957bb8ef-8ebd-4f16-8c28-254b741c5b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b" Nov 01 06:25:23.779566 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:23.779487 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nf49\" (UniqueName: \"kubernetes.io/projected/957bb8ef-8ebd-4f16-8c28-254b741c5b5e-kube-api-access-8nf49\") pod \"maas-api-revoke-sub-b77cc454-l867b\" (UID: \"957bb8ef-8ebd-4f16-8c28-254b741c5b5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b" Nov 01 06:25:23.828379 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:23.828348 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b" Nov 01 06:25:24.207982 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:25:24.207950 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957bb8ef_8ebd_4f16_8c28_254b741c5b5e.slice/crio-6aaf448756a78a4904a17e6b7742d3861c1c3b8678e95afc4999310d0289197d WatchSource:0}: Error finding container 6aaf448756a78a4904a17e6b7742d3861c1c3b8678e95afc4999310d0289197d: Status 404 returned error can't find the container with id 6aaf448756a78a4904a17e6b7742d3861c1c3b8678e95afc4999310d0289197d Nov 01 06:25:24.208643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:24.208417 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b"] Nov 01 06:25:25.012370 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:25.012333 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b" event={"ID":"957bb8ef-8ebd-4f16-8c28-254b741c5b5e","Type":"ContainerStarted","Data":"d6dd8535e01722485f3cebbe71613c13906faef60a65a1184da7d84aea17499a"} Nov 01 06:25:25.012522 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:25.012385 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b" event={"ID":"957bb8ef-8ebd-4f16-8c28-254b741c5b5e","Type":"ContainerStarted","Data":"6aaf448756a78a4904a17e6b7742d3861c1c3b8678e95afc4999310d0289197d"} Nov 01 06:25:25.125727 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:25.125670 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b" podStartSLOduration=2.125653399 podStartE2EDuration="2.125653399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:25.123940714 +0000 UTC m=+934.887740572" watchObservedRunningTime="2026-11-01 06:25:25.125653399 +0000 UTC m=+934.889453257" Nov 01 06:25:26.019708 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:26.019676 4385 generic.go:356] "Generic (PLEG): container finished" podID="957bb8ef-8ebd-4f16-8c28-254b741c5b5e" containerID="d6dd8535e01722485f3cebbe71613c13906faef60a65a1184da7d84aea17499a" exitCode=0 Nov 01 06:25:26.020191 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:26.019757 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b" event={"ID":"957bb8ef-8ebd-4f16-8c28-254b741c5b5e","Type":"ContainerDied","Data":"d6dd8535e01722485f3cebbe71613c13906faef60a65a1184da7d84aea17499a"} Nov 01 06:25:27.179400 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:27.179370 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b" Nov 01 06:25:27.259277 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:27.259239 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/957bb8ef-8ebd-4f16-8c28-254b741c5b5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/957bb8ef-8ebd-4f16-8c28-254b741c5b5e-maas-api-xks-ca\") pod \"957bb8ef-8ebd-4f16-8c28-254b741c5b5e\" (UID: \"957bb8ef-8ebd-4f16-8c28-254b741c5b5e\") " Nov 01 06:25:27.259448 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:27.259391 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/957bb8ef-8ebd-4f16-8c28-254b741c5b5e-kube-api-access-8nf49\" (UniqueName: \"kubernetes.io/projected/957bb8ef-8ebd-4f16-8c28-254b741c5b5e-kube-api-access-8nf49\") pod \"957bb8ef-8ebd-4f16-8c28-254b741c5b5e\" (UID: \"957bb8ef-8ebd-4f16-8c28-254b741c5b5e\") " Nov 01 06:25:27.259448 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:27.259428 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/957bb8ef-8ebd-4f16-8c28-254b741c5b5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/957bb8ef-8ebd-4f16-8c28-254b741c5b5e-maas-api-openshift-ca\") pod \"957bb8ef-8ebd-4f16-8c28-254b741c5b5e\" (UID: \"957bb8ef-8ebd-4f16-8c28-254b741c5b5e\") " Nov 01 06:25:27.259827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:27.259801 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957bb8ef-8ebd-4f16-8c28-254b741c5b5e-maas-api-openshift-ca" pod "957bb8ef-8ebd-4f16-8c28-254b741c5b5e" (UID: "957bb8ef-8ebd-4f16-8c28-254b741c5b5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:25:27.261514 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:27.261485 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/957bb8ef-8ebd-4f16-8c28-254b741c5b5e-maas-api-xks-ca" pod "957bb8ef-8ebd-4f16-8c28-254b741c5b5e" (UID: "957bb8ef-8ebd-4f16-8c28-254b741c5b5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:25:27.261667 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:27.261647 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957bb8ef-8ebd-4f16-8c28-254b741c5b5e-kube-api-access-8nf49" pod "957bb8ef-8ebd-4f16-8c28-254b741c5b5e" (UID: "957bb8ef-8ebd-4f16-8c28-254b741c5b5e"). InnerVolumeSpecName "kube-api-access-8nf49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:25:27.360331 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:27.360296 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/957bb8ef-8ebd-4f16-8c28-254b741c5b5e-maas-api-openshift-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:27.360331 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:27.360327 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/957bb8ef-8ebd-4f16-8c28-254b741c5b5e-maas-api-xks-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:27.360331 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:27.360338 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nf49\" (UniqueName: \"kubernetes.io/projected/957bb8ef-8ebd-4f16-8c28-254b741c5b5e-kube-api-access-8nf49\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:27.832606 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:27.832573 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:28.037088 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:28.037057 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b" Nov 01 06:25:28.037261 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:28.037086 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b" event={"ID":"957bb8ef-8ebd-4f16-8c28-254b741c5b5e","Type":"ContainerDied","Data":"6aaf448756a78a4904a17e6b7742d3861c1c3b8678e95afc4999310d0289197d"} Nov 01 06:25:28.037261 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:28.037139 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6aaf448756a78a4904a17e6b7742d3861c1c3b8678e95afc4999310d0289197d" Nov 01 06:25:32.846341 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:32.846258 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:33.762054 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:33.762006 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:36.791611 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:25:36.791572 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:25:37.655959 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:37.655920 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:39.249118 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:39.249081 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:39.821598 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:39.821558 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:40.390094 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:40.390060 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87f643dd-4nzbz"] Nov 01 06:25:40.390858 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:40.390819 4385 cpu_manager.go:414] "containerMap: removing container" podUID="957bb8ef-8ebd-4f16-8c28-254b741c5b5e" containerName="revoke-keys" Nov 01 06:25:40.390935 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:40.390861 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="957bb8ef-8ebd-4f16-8c28-254b741c5b5e" containerName="revoke-keys" Nov 01 06:25:40.391026 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:40.391012 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="957bb8ef-8ebd-4f16-8c28-254b741c5b5e" containerName="revoke-keys" Nov 01 06:25:40.398045 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:40.398009 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f643dd-4nzbz" Nov 01 06:25:40.447377 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:40.447344 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87f643dd-4nzbz"] Nov 01 06:25:40.493368 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:40.493341 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:25:40.493368 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:40.493359 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\"" Nov 01 06:25:40.514706 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:40.514674 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dl7l\" (UniqueName: \"kubernetes.io/projected/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19-kube-api-access-9dl7l\") pod \"maas-api-revoke-sub-87f643dd-4nzbz\" (UID: \"f986ea82-ba44-4bc9-9eeb-f9f434c6bf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f643dd-4nzbz" Nov 01 06:25:40.514898 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:40.514718 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87f643dd-4nzbz\" (UID: \"f986ea82-ba44-4bc9-9eeb-f9f434c6bf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f643dd-4nzbz" Nov 01 06:25:40.514958 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:40.514907 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87f643dd-4nzbz\" (UID: \"f986ea82-ba44-4bc9-9eeb-f9f434c6bf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f643dd-4nzbz" Nov 01 06:25:40.615587 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:40.615554 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dl7l\" (UniqueName: \"kubernetes.io/projected/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19-kube-api-access-9dl7l\") pod \"maas-api-revoke-sub-87f643dd-4nzbz\" (UID: \"f986ea82-ba44-4bc9-9eeb-f9f434c6bf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f643dd-4nzbz" Nov 01 06:25:40.615773 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:40.615601 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87f643dd-4nzbz\" (UID: \"f986ea82-ba44-4bc9-9eeb-f9f434c6bf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f643dd-4nzbz" Nov 01 06:25:40.615773 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:40.615707 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87f643dd-4nzbz\" (UID: \"f986ea82-ba44-4bc9-9eeb-f9f434c6bf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f643dd-4nzbz" Nov 01 06:25:40.616300 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:40.616281 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87f643dd-4nzbz\" (UID: \"f986ea82-ba44-4bc9-9eeb-f9f434c6bf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f643dd-4nzbz" Nov 01 06:25:40.618258 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:40.618233 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87f643dd-4nzbz\" (UID: \"f986ea82-ba44-4bc9-9eeb-f9f434c6bf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f643dd-4nzbz" Nov 01 06:25:40.671953 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:40.671878 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dl7l\" (UniqueName: \"kubernetes.io/projected/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19-kube-api-access-9dl7l\") pod \"maas-api-revoke-sub-87f643dd-4nzbz\" (UID: \"f986ea82-ba44-4bc9-9eeb-f9f434c6bf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f643dd-4nzbz" Nov 01 06:25:40.714109 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:40.714064 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f643dd-4nzbz" Nov 01 06:25:41.088364 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:25:41.088324 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf986ea82_ba44_4bc9_9eeb_f9f434c6bf19.slice/crio-4a7afe0a0e226495d69e1fb2fc78adc74123361e56256c44b0e25e94b5ef29ef WatchSource:0}: Error finding container 4a7afe0a0e226495d69e1fb2fc78adc74123361e56256c44b0e25e94b5ef29ef: Status 404 returned error can't find the container with id 4a7afe0a0e226495d69e1fb2fc78adc74123361e56256c44b0e25e94b5ef29ef Nov 01 06:25:41.088856 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:41.088807 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87f643dd-4nzbz"] Nov 01 06:25:41.124077 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:41.124045 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f643dd-4nzbz" event={"ID":"f986ea82-ba44-4bc9-9eeb-f9f434c6bf19","Type":"ContainerStarted","Data":"4a7afe0a0e226495d69e1fb2fc78adc74123361e56256c44b0e25e94b5ef29ef"} Nov 01 06:25:42.131462 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:42.131432 4385 generic.go:356] "Generic (PLEG): container finished" podID="f986ea82-ba44-4bc9-9eeb-f9f434c6bf19" containerID="1fcb2ca0f766c825333a8cf773a3fc2379221318c2c64aeacfce98475e674435" exitCode=0 Nov 01 06:25:42.131910 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:42.131575 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f643dd-4nzbz" event={"ID":"f986ea82-ba44-4bc9-9eeb-f9f434c6bf19","Type":"ContainerDied","Data":"1fcb2ca0f766c825333a8cf773a3fc2379221318c2c64aeacfce98475e674435"} Nov 01 06:25:42.347846 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:42.347805 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:43.013047 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:43.013017 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:43.281534 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:43.281456 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f643dd-4nzbz" Nov 01 06:25:43.360159 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:43.360127 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19-maas-api-openshift-ca\") pod \"f986ea82-ba44-4bc9-9eeb-f9f434c6bf19\" (UID: \"f986ea82-ba44-4bc9-9eeb-f9f434c6bf19\") " Nov 01 06:25:43.360370 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:43.360226 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19-kube-api-access-9dl7l\" (UniqueName: \"kubernetes.io/projected/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19-kube-api-access-9dl7l\") pod \"f986ea82-ba44-4bc9-9eeb-f9f434c6bf19\" (UID: \"f986ea82-ba44-4bc9-9eeb-f9f434c6bf19\") " Nov 01 06:25:43.360370 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:43.360254 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19-maas-api-xks-ca\") pod \"f986ea82-ba44-4bc9-9eeb-f9f434c6bf19\" (UID: \"f986ea82-ba44-4bc9-9eeb-f9f434c6bf19\") " Nov 01 06:25:43.360563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:43.360534 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19-maas-api-openshift-ca" pod "f986ea82-ba44-4bc9-9eeb-f9f434c6bf19" (UID: "f986ea82-ba44-4bc9-9eeb-f9f434c6bf19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:25:43.362490 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:43.362462 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19-maas-api-xks-ca" pod "f986ea82-ba44-4bc9-9eeb-f9f434c6bf19" (UID: "f986ea82-ba44-4bc9-9eeb-f9f434c6bf19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:25:43.362584 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:43.362486 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19-kube-api-access-9dl7l" pod "f986ea82-ba44-4bc9-9eeb-f9f434c6bf19" (UID: "f986ea82-ba44-4bc9-9eeb-f9f434c6bf19"). InnerVolumeSpecName "kube-api-access-9dl7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:25:43.461527 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:43.461488 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dl7l\" (UniqueName: \"kubernetes.io/projected/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19-kube-api-access-9dl7l\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:43.461527 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:43.461525 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19-maas-api-xks-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:43.461784 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:43.461552 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19-maas-api-openshift-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:43.811964 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:43.811874 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:44.122869 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:44.122803 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:44.145337 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:44.145313 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f643dd-4nzbz" Nov 01 06:25:44.145526 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:44.145346 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f643dd-4nzbz" event={"ID":"f986ea82-ba44-4bc9-9eeb-f9f434c6bf19","Type":"ContainerDied","Data":"4a7afe0a0e226495d69e1fb2fc78adc74123361e56256c44b0e25e94b5ef29ef"} Nov 01 06:25:44.145526 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:44.145401 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a7afe0a0e226495d69e1fb2fc78adc74123361e56256c44b0e25e94b5ef29ef" Nov 01 06:25:45.154148 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:45.154112 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:46.268121 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:46.268084 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:46.508562 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:46.508522 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:46.657473 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:46.657439 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:46.858206 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:46.858162 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:47.030266 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:47.030152 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:47.413346 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:47.413310 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:47.581861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:47.581813 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:47.736353 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:47.736258 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:48.114186 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:48.114138 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:48.263492 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:48.263457 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:52.250252 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:52.250214 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:52.454234 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:52.454200 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:55.323960 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:55.323908 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf59e74e-4c88q"] Nov 01 06:25:55.324574 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:55.324553 4385 cpu_manager.go:414] "containerMap: removing container" podUID="f986ea82-ba44-4bc9-9eeb-f9f434c6bf19" containerName="revoke-keys" Nov 01 06:25:55.324666 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:55.324576 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f986ea82-ba44-4bc9-9eeb-f9f434c6bf19" containerName="revoke-keys" Nov 01 06:25:55.324756 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:55.324742 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="f986ea82-ba44-4bc9-9eeb-f9f434c6bf19" containerName="revoke-keys" Nov 01 06:25:55.328062 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:55.328041 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59e74e-4c88q" Nov 01 06:25:55.383644 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:55.383608 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf59e74e-4c88q"] Nov 01 06:25:55.423470 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:55.423433 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381f6bdd-1c00-4f4a-bc28-f901428da687-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf59e74e-4c88q\" (UID: \"381f6bdd-1c00-4f4a-bc28-f901428da687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59e74e-4c88q" Nov 01 06:25:55.423651 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:55.423593 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9dhd\" (UniqueName: \"kubernetes.io/projected/381f6bdd-1c00-4f4a-bc28-f901428da687-kube-api-access-g9dhd\") pod \"maas-api-revoke-sub-cf59e74e-4c88q\" (UID: \"381f6bdd-1c00-4f4a-bc28-f901428da687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59e74e-4c88q" Nov 01 06:25:55.423709 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:55.423653 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381f6bdd-1c00-4f4a-bc28-f901428da687-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf59e74e-4c88q\" (UID: \"381f6bdd-1c00-4f4a-bc28-f901428da687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59e74e-4c88q" Nov 01 06:25:55.429115 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:55.429089 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:25:55.429254 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:55.429143 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\"" Nov 01 06:25:55.525052 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:55.525011 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9dhd\" (UniqueName: \"kubernetes.io/projected/381f6bdd-1c00-4f4a-bc28-f901428da687-kube-api-access-g9dhd\") pod \"maas-api-revoke-sub-cf59e74e-4c88q\" (UID: \"381f6bdd-1c00-4f4a-bc28-f901428da687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59e74e-4c88q" Nov 01 06:25:55.525261 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:55.525093 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381f6bdd-1c00-4f4a-bc28-f901428da687-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf59e74e-4c88q\" (UID: \"381f6bdd-1c00-4f4a-bc28-f901428da687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59e74e-4c88q" Nov 01 06:25:55.525261 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:55.525156 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381f6bdd-1c00-4f4a-bc28-f901428da687-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf59e74e-4c88q\" (UID: \"381f6bdd-1c00-4f4a-bc28-f901428da687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59e74e-4c88q" Nov 01 06:25:55.525891 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:55.525867 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381f6bdd-1c00-4f4a-bc28-f901428da687-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf59e74e-4c88q\" (UID: \"381f6bdd-1c00-4f4a-bc28-f901428da687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59e74e-4c88q" Nov 01 06:25:55.527869 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:55.527846 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381f6bdd-1c00-4f4a-bc28-f901428da687-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf59e74e-4c88q\" (UID: \"381f6bdd-1c00-4f4a-bc28-f901428da687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59e74e-4c88q" Nov 01 06:25:55.582120 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:55.582037 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9dhd\" (UniqueName: \"kubernetes.io/projected/381f6bdd-1c00-4f4a-bc28-f901428da687-kube-api-access-g9dhd\") pod \"maas-api-revoke-sub-cf59e74e-4c88q\" (UID: \"381f6bdd-1c00-4f4a-bc28-f901428da687\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59e74e-4c88q" Nov 01 06:25:55.641397 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:55.641357 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59e74e-4c88q" Nov 01 06:25:56.017466 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:25:56.017422 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381f6bdd_1c00_4f4a_bc28_f901428da687.slice/crio-a6de87f421cfecff62b631dd78f72d0260d1d0ed5baabf0034cb0144279e4527 WatchSource:0}: Error finding container a6de87f421cfecff62b631dd78f72d0260d1d0ed5baabf0034cb0144279e4527: Status 404 returned error can't find the container with id a6de87f421cfecff62b631dd78f72d0260d1d0ed5baabf0034cb0144279e4527 Nov 01 06:25:56.018093 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:56.018065 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf59e74e-4c88q"] Nov 01 06:25:56.231777 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:56.231740 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59e74e-4c88q" event={"ID":"381f6bdd-1c00-4f4a-bc28-f901428da687","Type":"ContainerStarted","Data":"a6de87f421cfecff62b631dd78f72d0260d1d0ed5baabf0034cb0144279e4527"} Nov 01 06:25:57.047168 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.047131 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d"] Nov 01 06:25:57.051052 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.051033 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.135969 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.135886 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-3ba2f3-openshift-default-dockercfg-nrd89\"" Nov 01 06:25:57.143980 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.143937 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.144161 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.143990 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.144161 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.144113 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.144297 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.144171 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.144297 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.144236 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg7jm\" (UniqueName: \"kubernetes.io/projected/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-kube-api-access-dg7jm\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.144382 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.144357 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.144432 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.144410 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.144432 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.144438 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.144602 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.144492 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.159600 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.159569 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d"] Nov 01 06:25:57.241717 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.241684 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:57.245190 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.245162 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.245342 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.245201 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg7jm\" (UniqueName: \"kubernetes.io/projected/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-kube-api-access-dg7jm\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.245342 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.245237 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.245342 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.245297 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.245342 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.245322 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.245552 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.245360 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.245552 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.245443 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.245552 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.245475 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.245552 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.245537 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.245741 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.245569 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.245794 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.245756 4385 generic.go:356] "Generic (PLEG): container finished" podID="381f6bdd-1c00-4f4a-bc28-f901428da687" containerID="eac34d58d932c859e3d4359cff37e9bc0edc9ec050b40c93ff97f9360d3825f9" exitCode=0 Nov 01 06:25:57.246046 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.246016 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.246144 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.246046 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59e74e-4c88q" event={"ID":"381f6bdd-1c00-4f4a-bc28-f901428da687","Type":"ContainerDied","Data":"eac34d58d932c859e3d4359cff37e9bc0edc9ec050b40c93ff97f9360d3825f9"} Nov 01 06:25:57.246249 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.246230 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.246313 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.246283 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.247183 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.247161 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.249174 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.249153 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.250162 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.250133 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.303477 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.303421 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.304122 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.304096 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg7jm\" (UniqueName: \"kubernetes.io/projected/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-kube-api-access-dg7jm\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.365626 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.365589 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:57.434357 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.432196 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:57.435690 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.435294 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw"] Nov 01 06:25:57.442138 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.442117 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.468120 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.468088 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw"] Nov 01 06:25:57.551643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.550257 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.551643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.550446 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.551643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.550547 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.551643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.550587 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.551643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.550621 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.551643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.550651 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tnsl\" (UniqueName: \"kubernetes.io/projected/ac1084eb-b2da-42cd-8376-f54f321c37b8-kube-api-access-9tnsl\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.551643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.550777 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1084eb-b2da-42cd-8376-f54f321c37b8-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.551643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.550865 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.551643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.550917 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.600879 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.600557 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:57.652407 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.652367 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.652604 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.652446 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.652604 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.652499 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.652604 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.652527 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.652604 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.652555 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.652604 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.652583 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tnsl\" (UniqueName: \"kubernetes.io/projected/ac1084eb-b2da-42cd-8376-f54f321c37b8-kube-api-access-9tnsl\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.652907 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.652617 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1084eb-b2da-42cd-8376-f54f321c37b8-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.652907 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.652668 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.652907 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.652720 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.652907 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.652756 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.653145 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.652939 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.653145 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.653026 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.653255 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.653205 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.653373 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.653350 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1084eb-b2da-42cd-8376-f54f321c37b8-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.655079 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.655057 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.655296 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.655278 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.710246 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.710165 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.710372 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.710327 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tnsl\" (UniqueName: \"kubernetes.io/projected/ac1084eb-b2da-42cd-8376-f54f321c37b8-kube-api-access-9tnsl\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.759245 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.759209 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:57.888986 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.888466 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d"] Nov 01 06:25:57.898685 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.896952 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:25:57.898685 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.897043 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:25:57.898685 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:57.897077 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:25:58.172923 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.172894 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw"] Nov 01 06:25:58.173910 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:25:58.173888 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1084eb_b2da_42cd_8376_f54f321c37b8.slice/crio-98920fe1e3d12bbceabc8b5ea11292b664d5983755a5fa4ad1b72c9212510226 WatchSource:0}: Error finding container 98920fe1e3d12bbceabc8b5ea11292b664d5983755a5fa4ad1b72c9212510226: Status 404 returned error can't find the container with id 98920fe1e3d12bbceabc8b5ea11292b664d5983755a5fa4ad1b72c9212510226 Nov 01 06:25:58.176957 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.176928 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:25:58.177038 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.176995 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:25:58.177038 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.177028 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:25:58.259253 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.259135 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" event={"ID":"ac1084eb-b2da-42cd-8376-f54f321c37b8","Type":"ContainerStarted","Data":"6f8fc53704ea10305d1efcfcd4e3badc98ca3287ea07cb0ce7afd1422cdc4900"} Nov 01 06:25:58.259253 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.259199 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" event={"ID":"ac1084eb-b2da-42cd-8376-f54f321c37b8","Type":"ContainerStarted","Data":"98920fe1e3d12bbceabc8b5ea11292b664d5983755a5fa4ad1b72c9212510226"} Nov 01 06:25:58.261895 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.261864 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" event={"ID":"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126","Type":"ContainerStarted","Data":"e3c4df095ed9da14695b370c3bea7d3ea226872d3744d6b2f570de1bc2ebda07"} Nov 01 06:25:58.262060 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.261913 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" event={"ID":"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126","Type":"ContainerStarted","Data":"89a5da19e916c70209f550bb7b878112d1847a8b031d74c55126f3c55fade2cb"} Nov 01 06:25:58.366519 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.366489 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:25:58.367620 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.367599 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 06:25:58.367682 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.367647 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" podUID="f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 06:25:58.381968 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.381683 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" podStartSLOduration=1.381663721 podStartE2EDuration="1.381663721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:58.379136908 +0000 UTC m=+968.142936777" watchObservedRunningTime="2026-11-01 06:25:58.381663721 +0000 UTC m=+968.145463579" Nov 01 06:25:58.435555 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.435518 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59e74e-4c88q" Nov 01 06:25:58.506885 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.506723 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" podStartSLOduration=1.506701324 podStartE2EDuration="1.506701324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:58.495777653 +0000 UTC m=+968.259577511" watchObservedRunningTime="2026-11-01 06:25:58.506701324 +0000 UTC m=+968.270501184" Nov 01 06:25:58.565046 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.565012 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/381f6bdd-1c00-4f4a-bc28-f901428da687-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381f6bdd-1c00-4f4a-bc28-f901428da687-maas-api-openshift-ca\") pod \"381f6bdd-1c00-4f4a-bc28-f901428da687\" (UID: \"381f6bdd-1c00-4f4a-bc28-f901428da687\") " Nov 01 06:25:58.565247 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.565229 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/381f6bdd-1c00-4f4a-bc28-f901428da687-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381f6bdd-1c00-4f4a-bc28-f901428da687-maas-api-xks-ca\") pod \"381f6bdd-1c00-4f4a-bc28-f901428da687\" (UID: \"381f6bdd-1c00-4f4a-bc28-f901428da687\") " Nov 01 06:25:58.565338 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.565290 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/381f6bdd-1c00-4f4a-bc28-f901428da687-kube-api-access-g9dhd\" (UniqueName: \"kubernetes.io/projected/381f6bdd-1c00-4f4a-bc28-f901428da687-kube-api-access-g9dhd\") pod \"381f6bdd-1c00-4f4a-bc28-f901428da687\" (UID: \"381f6bdd-1c00-4f4a-bc28-f901428da687\") " Nov 01 06:25:58.565417 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.565377 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381f6bdd-1c00-4f4a-bc28-f901428da687-maas-api-openshift-ca" pod "381f6bdd-1c00-4f4a-bc28-f901428da687" (UID: "381f6bdd-1c00-4f4a-bc28-f901428da687"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:25:58.565702 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.565639 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381f6bdd-1c00-4f4a-bc28-f901428da687-maas-api-openshift-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:58.568869 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.568818 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381f6bdd-1c00-4f4a-bc28-f901428da687-kube-api-access-g9dhd" pod "381f6bdd-1c00-4f4a-bc28-f901428da687" (UID: "381f6bdd-1c00-4f4a-bc28-f901428da687"). InnerVolumeSpecName "kube-api-access-g9dhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:25:58.569277 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.569233 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381f6bdd-1c00-4f4a-bc28-f901428da687-maas-api-xks-ca" pod "381f6bdd-1c00-4f4a-bc28-f901428da687" (UID: "381f6bdd-1c00-4f4a-bc28-f901428da687"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:25:58.666782 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.666707 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381f6bdd-1c00-4f4a-bc28-f901428da687-maas-api-xks-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:58.666782 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.666749 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9dhd\" (UniqueName: \"kubernetes.io/projected/381f6bdd-1c00-4f4a-bc28-f901428da687-kube-api-access-g9dhd\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:25:58.759409 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:58.759336 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:25:59.311877 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:59.307896 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59e74e-4c88q" Nov 01 06:25:59.314158 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:59.313377 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf59e74e-4c88q" event={"ID":"381f6bdd-1c00-4f4a-bc28-f901428da687","Type":"ContainerDied","Data":"a6de87f421cfecff62b631dd78f72d0260d1d0ed5baabf0034cb0144279e4527"} Nov 01 06:25:59.314158 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:59.313877 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6de87f421cfecff62b631dd78f72d0260d1d0ed5baabf0034cb0144279e4527" Nov 01 06:25:59.340455 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:59.340427 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:25:59.366543 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:59.366395 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 06:25:59.366543 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:59.366465 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" podUID="f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 06:25:59.762093 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:59.762044 4385 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 06:25:59.762279 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:25:59.762107 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" podUID="ac1084eb-b2da-42cd-8376-f54f321c37b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 06:26:00.366744 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:00.366710 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 06:26:00.367193 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:00.366763 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" podUID="f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 06:26:00.760267 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:00.760177 4385 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 06:26:00.760267 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:00.760228 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" podUID="ac1084eb-b2da-42cd-8376-f54f321c37b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 06:26:01.366497 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:01.366459 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 06:26:01.366667 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:01.366511 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" podUID="f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 06:26:01.760277 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:01.760187 4385 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 06:26:01.760277 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:01.760236 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" podUID="ac1084eb-b2da-42cd-8376-f54f321c37b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 06:26:02.036710 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:02.036624 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:26:02.366759 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:02.366723 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 06:26:02.366976 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:02.366782 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" podUID="f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 06:26:02.760548 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:02.760461 4385 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 06:26:02.760548 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:02.760529 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" podUID="ac1084eb-b2da-42cd-8376-f54f321c37b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 06:26:03.366922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:03.366894 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 06:26:03.367168 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:03.367136 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" podUID="f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 06:26:03.760056 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:03.759968 4385 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 06:26:03.760235 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:03.760197 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" podUID="ac1084eb-b2da-42cd-8376-f54f321c37b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 06:26:04.366255 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:04.366218 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 01 06:26:04.366684 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:04.366268 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" podUID="f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 01 06:26:04.760244 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:04.760157 4385 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 06:26:04.760244 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:04.760224 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" podUID="ac1084eb-b2da-42cd-8376-f54f321c37b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 06:26:05.370769 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:05.370739 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:26:05.371261 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:05.371242 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:26:05.372421 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:05.372400 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:26:05.875226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:05.875192 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:26:05.876105 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:05.876066 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:26:05.877959 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:05.877937 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 06:26:06.066280 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:06.066239 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s"] Nov 01 06:26:06.066739 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:06.066723 4385 cpu_manager.go:414] "containerMap: removing container" podUID="381f6bdd-1c00-4f4a-bc28-f901428da687" containerName="revoke-keys" Nov 01 06:26:06.066739 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:06.066738 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381f6bdd-1c00-4f4a-bc28-f901428da687" containerName="revoke-keys" Nov 01 06:26:06.066934 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:06.066848 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="381f6bdd-1c00-4f4a-bc28-f901428da687" containerName="revoke-keys" Nov 01 06:26:06.070167 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:06.070145 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s" Nov 01 06:26:06.120751 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:06.120720 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11580db4-78ae-4c6f-8cfe-68acd757b4c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5898253b-hgh8s\" (UID: \"11580db4-78ae-4c6f-8cfe-68acd757b4c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s" Nov 01 06:26:06.120751 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:06.120755 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bprt6\" (UniqueName: \"kubernetes.io/projected/11580db4-78ae-4c6f-8cfe-68acd757b4c2-kube-api-access-bprt6\") pod \"maas-api-revoke-sub-5898253b-hgh8s\" (UID: \"11580db4-78ae-4c6f-8cfe-68acd757b4c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s" Nov 01 06:26:06.121056 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:06.120787 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11580db4-78ae-4c6f-8cfe-68acd757b4c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5898253b-hgh8s\" (UID: \"11580db4-78ae-4c6f-8cfe-68acd757b4c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s" Nov 01 06:26:06.124881 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:06.124106 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:26:06.124881 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:06.124145 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\"" Nov 01 06:26:06.126138 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:06.126062 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s"] Nov 01 06:26:06.222159 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:06.222123 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11580db4-78ae-4c6f-8cfe-68acd757b4c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5898253b-hgh8s\" (UID: \"11580db4-78ae-4c6f-8cfe-68acd757b4c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s" Nov 01 06:26:06.222380 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:06.222360 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11580db4-78ae-4c6f-8cfe-68acd757b4c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5898253b-hgh8s\" (UID: \"11580db4-78ae-4c6f-8cfe-68acd757b4c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s" Nov 01 06:26:06.222447 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:06.222397 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bprt6\" (UniqueName: \"kubernetes.io/projected/11580db4-78ae-4c6f-8cfe-68acd757b4c2-kube-api-access-bprt6\") pod \"maas-api-revoke-sub-5898253b-hgh8s\" (UID: \"11580db4-78ae-4c6f-8cfe-68acd757b4c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s" Nov 01 06:26:06.223145 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:06.223107 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11580db4-78ae-4c6f-8cfe-68acd757b4c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5898253b-hgh8s\" (UID: \"11580db4-78ae-4c6f-8cfe-68acd757b4c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s" Nov 01 06:26:06.225193 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:06.225167 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11580db4-78ae-4c6f-8cfe-68acd757b4c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5898253b-hgh8s\" (UID: \"11580db4-78ae-4c6f-8cfe-68acd757b4c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s" Nov 01 06:26:06.278762 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:06.278724 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bprt6\" (UniqueName: \"kubernetes.io/projected/11580db4-78ae-4c6f-8cfe-68acd757b4c2-kube-api-access-bprt6\") pod \"maas-api-revoke-sub-5898253b-hgh8s\" (UID: \"11580db4-78ae-4c6f-8cfe-68acd757b4c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s" Nov 01 06:26:06.386698 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:06.386618 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s" Nov 01 06:26:06.772008 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:06.771969 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s"] Nov 01 06:26:06.772682 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:26:06.772655 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11580db4_78ae_4c6f_8cfe_68acd757b4c2.slice/crio-00657a04ab89c4a980b4d9807f8bcf4c4636718200fdd92a0689104eb54464a0 WatchSource:0}: Error finding container 00657a04ab89c4a980b4d9807f8bcf4c4636718200fdd92a0689104eb54464a0: Status 404 returned error can't find the container with id 00657a04ab89c4a980b4d9807f8bcf4c4636718200fdd92a0689104eb54464a0 Nov 01 06:26:06.789911 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:26:06.789873 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:26:06.946491 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:06.946448 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:26:07.384292 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:07.384198 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s" event={"ID":"11580db4-78ae-4c6f-8cfe-68acd757b4c2","Type":"ContainerStarted","Data":"c7d7a46bab1ec474aa8acf44fce2f8a62db47ef44f72ab22b6f0d362c4ab7d93"} Nov 01 06:26:07.384292 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:07.384262 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s" event={"ID":"11580db4-78ae-4c6f-8cfe-68acd757b4c2","Type":"ContainerStarted","Data":"00657a04ab89c4a980b4d9807f8bcf4c4636718200fdd92a0689104eb54464a0"} Nov 01 06:26:07.498950 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:07.498895 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s" podStartSLOduration=1.498880003 podStartE2EDuration="1.498880003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:26:07.495801874 +0000 UTC m=+977.259601731" watchObservedRunningTime="2026-11-01 06:26:07.498880003 +0000 UTC m=+977.262679859" Nov 01 06:26:08.439207 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:08.439165 4385 generic.go:356] "Generic (PLEG): container finished" podID="11580db4-78ae-4c6f-8cfe-68acd757b4c2" containerID="c7d7a46bab1ec474aa8acf44fce2f8a62db47ef44f72ab22b6f0d362c4ab7d93" exitCode=0 Nov 01 06:26:08.439424 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:08.439346 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s" event={"ID":"11580db4-78ae-4c6f-8cfe-68acd757b4c2","Type":"ContainerDied","Data":"c7d7a46bab1ec474aa8acf44fce2f8a62db47ef44f72ab22b6f0d362c4ab7d93"} Nov 01 06:26:09.618341 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:09.618314 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s" Nov 01 06:26:09.673192 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:09.673154 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11580db4-78ae-4c6f-8cfe-68acd757b4c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11580db4-78ae-4c6f-8cfe-68acd757b4c2-maas-api-xks-ca\") pod \"11580db4-78ae-4c6f-8cfe-68acd757b4c2\" (UID: \"11580db4-78ae-4c6f-8cfe-68acd757b4c2\") " Nov 01 06:26:09.673366 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:09.673247 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11580db4-78ae-4c6f-8cfe-68acd757b4c2-kube-api-access-bprt6\" (UniqueName: \"kubernetes.io/projected/11580db4-78ae-4c6f-8cfe-68acd757b4c2-kube-api-access-bprt6\") pod \"11580db4-78ae-4c6f-8cfe-68acd757b4c2\" (UID: \"11580db4-78ae-4c6f-8cfe-68acd757b4c2\") " Nov 01 06:26:09.673453 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:09.673412 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11580db4-78ae-4c6f-8cfe-68acd757b4c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11580db4-78ae-4c6f-8cfe-68acd757b4c2-maas-api-openshift-ca\") pod \"11580db4-78ae-4c6f-8cfe-68acd757b4c2\" (UID: \"11580db4-78ae-4c6f-8cfe-68acd757b4c2\") " Nov 01 06:26:09.673828 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:09.673792 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11580db4-78ae-4c6f-8cfe-68acd757b4c2-maas-api-openshift-ca" pod "11580db4-78ae-4c6f-8cfe-68acd757b4c2" (UID: "11580db4-78ae-4c6f-8cfe-68acd757b4c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:26:09.675635 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:09.675605 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11580db4-78ae-4c6f-8cfe-68acd757b4c2-maas-api-xks-ca" pod "11580db4-78ae-4c6f-8cfe-68acd757b4c2" (UID: "11580db4-78ae-4c6f-8cfe-68acd757b4c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:26:09.675635 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:09.675621 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11580db4-78ae-4c6f-8cfe-68acd757b4c2-kube-api-access-bprt6" pod "11580db4-78ae-4c6f-8cfe-68acd757b4c2" (UID: "11580db4-78ae-4c6f-8cfe-68acd757b4c2"). InnerVolumeSpecName "kube-api-access-bprt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:26:09.774636 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:09.774556 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11580db4-78ae-4c6f-8cfe-68acd757b4c2-maas-api-openshift-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:09.774636 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:09.774586 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11580db4-78ae-4c6f-8cfe-68acd757b4c2-maas-api-xks-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:09.774636 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:09.774597 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bprt6\" (UniqueName: \"kubernetes.io/projected/11580db4-78ae-4c6f-8cfe-68acd757b4c2-kube-api-access-bprt6\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:10.453559 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:10.453535 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s" Nov 01 06:26:10.453744 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:10.453571 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s" event={"ID":"11580db4-78ae-4c6f-8cfe-68acd757b4c2","Type":"ContainerDied","Data":"00657a04ab89c4a980b4d9807f8bcf4c4636718200fdd92a0689104eb54464a0"} Nov 01 06:26:10.453744 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:10.453617 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00657a04ab89c4a980b4d9807f8bcf4c4636718200fdd92a0689104eb54464a0" Nov 01 06:26:11.629018 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:11.628984 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:26:16.412593 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:16.412541 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:26:21.220258 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:21.220201 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:26:22.302974 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:22.302937 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h"] Nov 01 06:26:22.303440 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:22.303427 4385 cpu_manager.go:414] "containerMap: removing container" podUID="11580db4-78ae-4c6f-8cfe-68acd757b4c2" containerName="revoke-keys" Nov 01 06:26:22.303482 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:22.303440 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11580db4-78ae-4c6f-8cfe-68acd757b4c2" containerName="revoke-keys" Nov 01 06:26:22.303535 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:22.303526 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="11580db4-78ae-4c6f-8cfe-68acd757b4c2" containerName="revoke-keys" Nov 01 06:26:22.308061 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:22.308042 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h" Nov 01 06:26:22.362256 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:22.362223 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h"] Nov 01 06:26:22.407150 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:22.407117 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:26:22.407313 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:22.407228 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\"" Nov 01 06:26:22.427964 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:22.427930 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfv9l\" (UniqueName: \"kubernetes.io/projected/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84-kube-api-access-zfv9l\") pod \"maas-api-revoke-sub-81031788-qdj4h\" (UID: \"0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h" Nov 01 06:26:22.428143 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:22.428051 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81031788-qdj4h\" (UID: \"0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h" Nov 01 06:26:22.428143 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:22.428114 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81031788-qdj4h\" (UID: \"0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h" Nov 01 06:26:22.529155 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:22.529121 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfv9l\" (UniqueName: \"kubernetes.io/projected/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84-kube-api-access-zfv9l\") pod \"maas-api-revoke-sub-81031788-qdj4h\" (UID: \"0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h" Nov 01 06:26:22.529353 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:22.529205 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81031788-qdj4h\" (UID: \"0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h" Nov 01 06:26:22.529353 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:22.529269 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81031788-qdj4h\" (UID: \"0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h" Nov 01 06:26:22.530006 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:22.529980 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81031788-qdj4h\" (UID: \"0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h" Nov 01 06:26:22.531694 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:22.531676 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81031788-qdj4h\" (UID: \"0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h" Nov 01 06:26:22.586949 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:22.586917 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfv9l\" (UniqueName: \"kubernetes.io/projected/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84-kube-api-access-zfv9l\") pod \"maas-api-revoke-sub-81031788-qdj4h\" (UID: \"0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h" Nov 01 06:26:22.621490 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:22.621459 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h" Nov 01 06:26:23.006355 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:23.006332 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h"] Nov 01 06:26:23.006797 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:26:23.006775 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf42b85_afb1_4d18_ae1a_9d73f1cbaf84.slice/crio-f706d2f26638c37449ef986c4b8a6fa600ad31e5d84ea07d03f32750826d280d WatchSource:0}: Error finding container f706d2f26638c37449ef986c4b8a6fa600ad31e5d84ea07d03f32750826d280d: Status 404 returned error can't find the container with id f706d2f26638c37449ef986c4b8a6fa600ad31e5d84ea07d03f32750826d280d Nov 01 06:26:23.572654 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:23.572551 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h" event={"ID":"0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84","Type":"ContainerStarted","Data":"6460215845cc1532d8c8c0f84ad10f8de8f79e4139aeb288d5a7e9a09f14166f"} Nov 01 06:26:23.572654 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:23.572620 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h" event={"ID":"0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84","Type":"ContainerStarted","Data":"f706d2f26638c37449ef986c4b8a6fa600ad31e5d84ea07d03f32750826d280d"} Nov 01 06:26:23.684824 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:23.684762 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h" podStartSLOduration=1.684748173 podStartE2EDuration="1.684748173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:26:23.684164387 +0000 UTC m=+993.447964245" watchObservedRunningTime="2026-11-01 06:26:23.684748173 +0000 UTC m=+993.448548040" Nov 01 06:26:24.583388 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:24.583351 4385 generic.go:356] "Generic (PLEG): container finished" podID="0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84" containerID="6460215845cc1532d8c8c0f84ad10f8de8f79e4139aeb288d5a7e9a09f14166f" exitCode=0 Nov 01 06:26:24.583822 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:24.583451 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h" event={"ID":"0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84","Type":"ContainerDied","Data":"6460215845cc1532d8c8c0f84ad10f8de8f79e4139aeb288d5a7e9a09f14166f"} Nov 01 06:26:25.780204 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:25.780169 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:26:25.782195 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:25.782172 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h" Nov 01 06:26:25.872624 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:25.872531 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84-kube-api-access-zfv9l\" (UniqueName: \"kubernetes.io/projected/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84-kube-api-access-zfv9l\") pod \"0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84\" (UID: \"0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84\") " Nov 01 06:26:25.873134 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:25.873078 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84-maas-api-openshift-ca\") pod \"0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84\" (UID: \"0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84\") " Nov 01 06:26:25.875035 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:25.873846 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84-maas-api-openshift-ca" pod "0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84" (UID: "0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:26:25.875035 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:25.874995 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84-maas-api-xks-ca\") pod \"0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84\" (UID: \"0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84\") " Nov 01 06:26:25.875534 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:25.875511 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84-maas-api-openshift-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:25.877468 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:25.877438 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84-maas-api-xks-ca" pod "0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84" (UID: "0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:26:25.877977 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:25.877954 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84-kube-api-access-zfv9l" pod "0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84" (UID: "0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84"). InnerVolumeSpecName "kube-api-access-zfv9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:26:25.944689 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:25.944650 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:26:25.977568 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:25.977525 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfv9l\" (UniqueName: \"kubernetes.io/projected/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84-kube-api-access-zfv9l\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:25.977568 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:25.977554 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84-maas-api-xks-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:26.131066 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:26.130985 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:26:26.602246 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:26.602217 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h" Nov 01 06:26:26.602437 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:26.602223 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h" event={"ID":"0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84","Type":"ContainerDied","Data":"f706d2f26638c37449ef986c4b8a6fa600ad31e5d84ea07d03f32750826d280d"} Nov 01 06:26:26.602437 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:26.602348 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f706d2f26638c37449ef986c4b8a6fa600ad31e5d84ea07d03f32750826d280d" Nov 01 06:26:26.928344 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:26.928307 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz"] Nov 01 06:26:26.928994 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:26.928974 4385 cpu_manager.go:414] "containerMap: removing container" podUID="0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84" containerName="revoke-keys" Nov 01 06:26:26.929095 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:26.928996 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84" containerName="revoke-keys" Nov 01 06:26:26.929153 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:26.929127 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84" containerName="revoke-keys" Nov 01 06:26:26.932693 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:26.932672 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:26.987303 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:26.987273 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-8ff537-openshift-default-dockercfg-sbjxb\"" Nov 01 06:26:26.992550 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:26.992521 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz"] Nov 01 06:26:27.091736 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.091690 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.091964 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.091748 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc38c4e-77a6-4175-9033-73067cc44525-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.091964 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.091853 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc4l5\" (UniqueName: \"kubernetes.io/projected/edc38c4e-77a6-4175-9033-73067cc44525-kube-api-access-qc4l5\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.091964 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.091875 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc38c4e-77a6-4175-9033-73067cc44525-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.091964 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.091895 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.091964 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.091917 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.092249 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.092056 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.092249 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.092081 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc38c4e-77a6-4175-9033-73067cc44525-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.092249 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.092119 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.193261 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.193162 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.193261 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.193212 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc38c4e-77a6-4175-9033-73067cc44525-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.193503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.193265 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.193503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.193316 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.193503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.193377 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc38c4e-77a6-4175-9033-73067cc44525-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.193503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.193455 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc4l5\" (UniqueName: \"kubernetes.io/projected/edc38c4e-77a6-4175-9033-73067cc44525-kube-api-access-qc4l5\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.193503 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.193487 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc38c4e-77a6-4175-9033-73067cc44525-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.193868 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.193518 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.193868 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.193555 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.193868 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.193671 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.193868 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.193694 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.194097 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.193939 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.194153 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.194117 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.194153 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.194145 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc38c4e-77a6-4175-9033-73067cc44525-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.196238 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.196213 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.196458 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.196436 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc38c4e-77a6-4175-9033-73067cc44525-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.250026 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.249996 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc38c4e-77a6-4175-9033-73067cc44525-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.250381 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.250364 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc4l5\" (UniqueName: \"kubernetes.io/projected/edc38c4e-77a6-4175-9033-73067cc44525-kube-api-access-qc4l5\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:27.546525 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:27.546435 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:28.021614 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:28.021580 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz"] Nov 01 06:26:28.022073 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:26:28.021691 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc38c4e_77a6_4175_9033_73067cc44525.slice/crio-4874e056663ef03274aa736a3542e8a4dd04caaaada26411b51d205559cce372 WatchSource:0}: Error finding container 4874e056663ef03274aa736a3542e8a4dd04caaaada26411b51d205559cce372: Status 404 returned error can't find the container with id 4874e056663ef03274aa736a3542e8a4dd04caaaada26411b51d205559cce372 Nov 01 06:26:28.024355 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:28.024328 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:26:28.024431 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:28.024386 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:26:28.024431 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:28.024414 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:26:28.647857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:28.641934 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" event={"ID":"edc38c4e-77a6-4175-9033-73067cc44525","Type":"ContainerStarted","Data":"155fc221a054f53da69cdb9d9ae8056c58ad4c78bb1396a7df063c133c17a362"} Nov 01 06:26:28.647857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:28.642008 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" event={"ID":"edc38c4e-77a6-4175-9033-73067cc44525","Type":"ContainerStarted","Data":"4874e056663ef03274aa736a3542e8a4dd04caaaada26411b51d205559cce372"} Nov 01 06:26:28.846791 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:28.846742 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" podStartSLOduration=2.846723111 podStartE2EDuration="2.846723111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:26:28.814861068 +0000 UTC m=+998.578660919" watchObservedRunningTime="2026-11-01 06:26:28.846723111 +0000 UTC m=+998.610522968" Nov 01 06:26:29.547017 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:29.546976 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:29.551983 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:29.551955 4385 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 06:26:29.552206 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:29.552170 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" podUID="edc38c4e-77a6-4175-9033-73067cc44525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 06:26:30.550588 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:30.550549 4385 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 06:26:30.551227 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:30.551197 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" podUID="edc38c4e-77a6-4175-9033-73067cc44525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 06:26:30.752860 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:30.749583 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:26:31.547604 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:31.547562 4385 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 06:26:31.547799 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:31.547620 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" podUID="edc38c4e-77a6-4175-9033-73067cc44525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 06:26:32.547475 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:32.547429 4385 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 06:26:32.548002 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:32.547492 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" podUID="edc38c4e-77a6-4175-9033-73067cc44525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 06:26:33.547419 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:33.547385 4385 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 06:26:33.547601 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:33.547437 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" podUID="edc38c4e-77a6-4175-9033-73067cc44525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 06:26:34.430853 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:34.430800 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:26:34.547359 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:34.547319 4385 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 06:26:34.547543 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:34.547393 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" podUID="edc38c4e-77a6-4175-9033-73067cc44525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 06:26:34.573404 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:34.573369 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:26:35.636417 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:35.636381 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:26:36.029980 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:36.029893 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:36.030387 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:36.030363 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:36.031400 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:36.031378 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 06:26:36.780671 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:26:36.780639 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:26:39.581119 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:39.581081 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:26:39.841534 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:39.841453 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df43ec64-gvm76"] Nov 01 06:26:39.845383 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:39.845365 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df43ec64-gvm76" Nov 01 06:26:39.896674 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:39.896629 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12175cf8-da78-4493-8764-7d1f189e006d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df43ec64-gvm76\" (UID: \"12175cf8-da78-4493-8764-7d1f189e006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df43ec64-gvm76" Nov 01 06:26:39.896825 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:39.896689 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4rr6\" (UniqueName: \"kubernetes.io/projected/12175cf8-da78-4493-8764-7d1f189e006d-kube-api-access-w4rr6\") pod \"maas-api-revoke-sub-df43ec64-gvm76\" (UID: \"12175cf8-da78-4493-8764-7d1f189e006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df43ec64-gvm76" Nov 01 06:26:39.896998 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:39.896978 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12175cf8-da78-4493-8764-7d1f189e006d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df43ec64-gvm76\" (UID: \"12175cf8-da78-4493-8764-7d1f189e006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df43ec64-gvm76" Nov 01 06:26:39.902871 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:39.902821 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df43ec64-gvm76"] Nov 01 06:26:39.945857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:39.945810 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:26:39.945857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:39.945856 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\"" Nov 01 06:26:39.997583 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:39.997554 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4rr6\" (UniqueName: \"kubernetes.io/projected/12175cf8-da78-4493-8764-7d1f189e006d-kube-api-access-w4rr6\") pod \"maas-api-revoke-sub-df43ec64-gvm76\" (UID: \"12175cf8-da78-4493-8764-7d1f189e006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df43ec64-gvm76" Nov 01 06:26:39.997758 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:39.997677 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12175cf8-da78-4493-8764-7d1f189e006d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df43ec64-gvm76\" (UID: \"12175cf8-da78-4493-8764-7d1f189e006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df43ec64-gvm76" Nov 01 06:26:39.997758 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:39.997714 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12175cf8-da78-4493-8764-7d1f189e006d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df43ec64-gvm76\" (UID: \"12175cf8-da78-4493-8764-7d1f189e006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df43ec64-gvm76" Nov 01 06:26:39.998332 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:39.998314 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12175cf8-da78-4493-8764-7d1f189e006d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df43ec64-gvm76\" (UID: \"12175cf8-da78-4493-8764-7d1f189e006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df43ec64-gvm76" Nov 01 06:26:40.000336 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:40.000316 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12175cf8-da78-4493-8764-7d1f189e006d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df43ec64-gvm76\" (UID: \"12175cf8-da78-4493-8764-7d1f189e006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df43ec64-gvm76" Nov 01 06:26:40.053680 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:40.053646 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4rr6\" (UniqueName: \"kubernetes.io/projected/12175cf8-da78-4493-8764-7d1f189e006d-kube-api-access-w4rr6\") pod \"maas-api-revoke-sub-df43ec64-gvm76\" (UID: \"12175cf8-da78-4493-8764-7d1f189e006d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df43ec64-gvm76" Nov 01 06:26:40.158954 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:40.158865 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df43ec64-gvm76" Nov 01 06:26:40.330153 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:40.330120 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:26:40.537068 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:26:40.537038 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12175cf8_da78_4493_8764_7d1f189e006d.slice/crio-88590d39a6ec0bb36641c0444fef8aaba09b23900f3441e3b00522fb0d83e32b WatchSource:0}: Error finding container 88590d39a6ec0bb36641c0444fef8aaba09b23900f3441e3b00522fb0d83e32b: Status 404 returned error can't find the container with id 88590d39a6ec0bb36641c0444fef8aaba09b23900f3441e3b00522fb0d83e32b Nov 01 06:26:40.537209 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:40.537176 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df43ec64-gvm76"] Nov 01 06:26:40.789803 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:40.789768 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df43ec64-gvm76" event={"ID":"12175cf8-da78-4493-8764-7d1f189e006d","Type":"ContainerStarted","Data":"88590d39a6ec0bb36641c0444fef8aaba09b23900f3441e3b00522fb0d83e32b"} Nov 01 06:26:41.797750 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:41.797717 4385 generic.go:356] "Generic (PLEG): container finished" podID="12175cf8-da78-4493-8764-7d1f189e006d" containerID="13b2cc213c4a7619e34edde04b3f3328818b3d51e22326dfc26ad45f96c7431a" exitCode=0 Nov 01 06:26:41.798161 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:41.797804 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df43ec64-gvm76" event={"ID":"12175cf8-da78-4493-8764-7d1f189e006d","Type":"ContainerDied","Data":"13b2cc213c4a7619e34edde04b3f3328818b3d51e22326dfc26ad45f96c7431a"} Nov 01 06:26:42.950863 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:42.950811 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df43ec64-gvm76" Nov 01 06:26:43.030599 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:43.030573 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12175cf8-da78-4493-8764-7d1f189e006d-kube-api-access-w4rr6\" (UniqueName: \"kubernetes.io/projected/12175cf8-da78-4493-8764-7d1f189e006d-kube-api-access-w4rr6\") pod \"12175cf8-da78-4493-8764-7d1f189e006d\" (UID: \"12175cf8-da78-4493-8764-7d1f189e006d\") " Nov 01 06:26:43.030774 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:43.030631 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12175cf8-da78-4493-8764-7d1f189e006d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12175cf8-da78-4493-8764-7d1f189e006d-maas-api-xks-ca\") pod \"12175cf8-da78-4493-8764-7d1f189e006d\" (UID: \"12175cf8-da78-4493-8764-7d1f189e006d\") " Nov 01 06:26:43.030774 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:43.030701 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12175cf8-da78-4493-8764-7d1f189e006d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12175cf8-da78-4493-8764-7d1f189e006d-maas-api-openshift-ca\") pod \"12175cf8-da78-4493-8764-7d1f189e006d\" (UID: \"12175cf8-da78-4493-8764-7d1f189e006d\") " Nov 01 06:26:43.031107 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:43.031085 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12175cf8-da78-4493-8764-7d1f189e006d-maas-api-openshift-ca" pod "12175cf8-da78-4493-8764-7d1f189e006d" (UID: "12175cf8-da78-4493-8764-7d1f189e006d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:26:43.032903 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:43.032886 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12175cf8-da78-4493-8764-7d1f189e006d-kube-api-access-w4rr6" pod "12175cf8-da78-4493-8764-7d1f189e006d" (UID: "12175cf8-da78-4493-8764-7d1f189e006d"). InnerVolumeSpecName "kube-api-access-w4rr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:26:43.032975 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:43.032898 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12175cf8-da78-4493-8764-7d1f189e006d-maas-api-xks-ca" pod "12175cf8-da78-4493-8764-7d1f189e006d" (UID: "12175cf8-da78-4493-8764-7d1f189e006d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:26:43.132536 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:43.132497 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12175cf8-da78-4493-8764-7d1f189e006d-maas-api-xks-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:43.132740 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:43.132725 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12175cf8-da78-4493-8764-7d1f189e006d-maas-api-openshift-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:43.132886 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:43.132870 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4rr6\" (UniqueName: \"kubernetes.io/projected/12175cf8-da78-4493-8764-7d1f189e006d-kube-api-access-w4rr6\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:43.811552 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:43.811518 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df43ec64-gvm76" Nov 01 06:26:43.811552 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:43.811520 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df43ec64-gvm76" event={"ID":"12175cf8-da78-4493-8764-7d1f189e006d","Type":"ContainerDied","Data":"88590d39a6ec0bb36641c0444fef8aaba09b23900f3441e3b00522fb0d83e32b"} Nov 01 06:26:43.811775 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:43.811567 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88590d39a6ec0bb36641c0444fef8aaba09b23900f3441e3b00522fb0d83e32b" Nov 01 06:26:45.139946 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:45.139899 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:26:46.419358 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:46.419323 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs"] Nov 01 06:26:46.435396 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:46.435359 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:26:47.052488 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:47.052460 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2"] Nov 01 06:26:47.052697 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:47.052675 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" podUID="753c422d-48cc-40ee-8673-208ed65fd311" containerName="istio-proxy" containerID="cri-o://bec44a5ed80eb8f4f920f6045eb96c59b456fd20f626e5b4ea45d3b3187beb28" gracePeriod=30 Nov 01 06:26:51.115416 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:51.115379 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:26:52.323316 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.323262 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:26:52.380668 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.380628 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/753c422d-48cc-40ee-8673-208ed65fd311-istio-token\" (UniqueName: \"kubernetes.io/projected/753c422d-48cc-40ee-8673-208ed65fd311-istio-token\") pod \"753c422d-48cc-40ee-8673-208ed65fd311\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " Nov 01 06:26:52.380863 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.380685 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-credential-socket\") pod \"753c422d-48cc-40ee-8673-208ed65fd311\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " Nov 01 06:26:52.380863 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.380733 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/753c422d-48cc-40ee-8673-208ed65fd311-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/753c422d-48cc-40ee-8673-208ed65fd311-istiod-ca-cert\") pod \"753c422d-48cc-40ee-8673-208ed65fd311\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " Nov 01 06:26:52.380863 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.380761 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-workload-socket\") pod \"753c422d-48cc-40ee-8673-208ed65fd311\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " Nov 01 06:26:52.380863 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.380794 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-istio-envoy\") pod \"753c422d-48cc-40ee-8673-208ed65fd311\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " Nov 01 06:26:52.380863 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.380853 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-workload-certs\") pod \"753c422d-48cc-40ee-8673-208ed65fd311\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " Nov 01 06:26:52.381140 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.380912 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/753c422d-48cc-40ee-8673-208ed65fd311-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/753c422d-48cc-40ee-8673-208ed65fd311-istio-podinfo\") pod \"753c422d-48cc-40ee-8673-208ed65fd311\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " Nov 01 06:26:52.381140 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.381025 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/753c422d-48cc-40ee-8673-208ed65fd311-kube-api-access-dcftq\" (UniqueName: \"kubernetes.io/projected/753c422d-48cc-40ee-8673-208ed65fd311-kube-api-access-dcftq\") pod \"753c422d-48cc-40ee-8673-208ed65fd311\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " Nov 01 06:26:52.381140 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.381037 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-credential-socket" pod "753c422d-48cc-40ee-8673-208ed65fd311" (UID: "753c422d-48cc-40ee-8673-208ed65fd311"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:26:52.381140 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.381139 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-istio-data\") pod \"753c422d-48cc-40ee-8673-208ed65fd311\" (UID: \"753c422d-48cc-40ee-8673-208ed65fd311\") " Nov 01 06:26:52.381351 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.381285 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-workload-certs" pod "753c422d-48cc-40ee-8673-208ed65fd311" (UID: "753c422d-48cc-40ee-8673-208ed65fd311"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:26:52.381514 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.381491 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-istio-data" pod "753c422d-48cc-40ee-8673-208ed65fd311" (UID: "753c422d-48cc-40ee-8673-208ed65fd311"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:26:52.381682 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.381658 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-workload-socket" pod "753c422d-48cc-40ee-8673-208ed65fd311" (UID: "753c422d-48cc-40ee-8673-208ed65fd311"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:26:52.381794 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.381782 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:52.381937 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.381802 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:52.381937 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.381817 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:52.381937 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.381846 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:52.382122 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.382066 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/753c422d-48cc-40ee-8673-208ed65fd311-istiod-ca-cert" pod "753c422d-48cc-40ee-8673-208ed65fd311" (UID: "753c422d-48cc-40ee-8673-208ed65fd311"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:26:52.384329 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.384209 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/753c422d-48cc-40ee-8673-208ed65fd311-kube-api-access-dcftq" pod "753c422d-48cc-40ee-8673-208ed65fd311" (UID: "753c422d-48cc-40ee-8673-208ed65fd311"). InnerVolumeSpecName "kube-api-access-dcftq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:26:52.384329 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.384259 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/753c422d-48cc-40ee-8673-208ed65fd311-istio-podinfo" pod "753c422d-48cc-40ee-8673-208ed65fd311" (UID: "753c422d-48cc-40ee-8673-208ed65fd311"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:26:52.384329 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.384275 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/753c422d-48cc-40ee-8673-208ed65fd311-istio-token" pod "753c422d-48cc-40ee-8673-208ed65fd311" (UID: "753c422d-48cc-40ee-8673-208ed65fd311"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:26:52.384570 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.384345 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-istio-envoy" pod "753c422d-48cc-40ee-8673-208ed65fd311" (UID: "753c422d-48cc-40ee-8673-208ed65fd311"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:26:52.483344 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.483301 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/753c422d-48cc-40ee-8673-208ed65fd311-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:52.483344 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.483332 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/753c422d-48cc-40ee-8673-208ed65fd311-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:52.483344 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.483342 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/753c422d-48cc-40ee-8673-208ed65fd311-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:52.483344 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.483352 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/753c422d-48cc-40ee-8673-208ed65fd311-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:52.483681 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.483361 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcftq\" (UniqueName: \"kubernetes.io/projected/753c422d-48cc-40ee-8673-208ed65fd311-kube-api-access-dcftq\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:52.899554 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.899521 4385 generic.go:356] "Generic (PLEG): container finished" podID="753c422d-48cc-40ee-8673-208ed65fd311" containerID="bec44a5ed80eb8f4f920f6045eb96c59b456fd20f626e5b4ea45d3b3187beb28" exitCode=0 Nov 01 06:26:52.899777 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.899578 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" Nov 01 06:26:52.899777 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.899644 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" event={"ID":"753c422d-48cc-40ee-8673-208ed65fd311","Type":"ContainerDied","Data":"bec44a5ed80eb8f4f920f6045eb96c59b456fd20f626e5b4ea45d3b3187beb28"} Nov 01 06:26:52.899777 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.899708 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2" event={"ID":"753c422d-48cc-40ee-8673-208ed65fd311","Type":"ContainerDied","Data":"1279c3c772decb9828e8bb97a9198bd25553a8a32f224178afcda6a1b581cde3"} Nov 01 06:26:52.899777 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.899741 4385 scope.go:122] "RemoveContainer" containerID="bec44a5ed80eb8f4f920f6045eb96c59b456fd20f626e5b4ea45d3b3187beb28" Nov 01 06:26:52.912471 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.912450 4385 scope.go:122] "RemoveContainer" containerID="bec44a5ed80eb8f4f920f6045eb96c59b456fd20f626e5b4ea45d3b3187beb28" Nov 01 06:26:52.912768 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:26:52.912743 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec44a5ed80eb8f4f920f6045eb96c59b456fd20f626e5b4ea45d3b3187beb28\": container with ID starting with bec44a5ed80eb8f4f920f6045eb96c59b456fd20f626e5b4ea45d3b3187beb28 not found: ID does not exist" containerID="bec44a5ed80eb8f4f920f6045eb96c59b456fd20f626e5b4ea45d3b3187beb28" Nov 01 06:26:52.912906 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:52.912775 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec44a5ed80eb8f4f920f6045eb96c59b456fd20f626e5b4ea45d3b3187beb28"} err="failed to get container status \"bec44a5ed80eb8f4f920f6045eb96c59b456fd20f626e5b4ea45d3b3187beb28\": rpc error: code = NotFound desc = could not find container \"bec44a5ed80eb8f4f920f6045eb96c59b456fd20f626e5b4ea45d3b3187beb28\": container with ID starting with bec44a5ed80eb8f4f920f6045eb96c59b456fd20f626e5b4ea45d3b3187beb28 not found: ID does not exist" Nov 01 06:26:53.065200 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:53.065168 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2"] Nov 01 06:26:53.114826 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:53.114790 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r5wn2"] Nov 01 06:26:53.426886 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:53.426853 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:26:53.571183 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:53.571146 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:26:54.383970 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:54.383936 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="753c422d-48cc-40ee-8673-208ed65fd311" path="/var/lib/kubelet/pods/753c422d-48cc-40ee-8673-208ed65fd311/volumes" Nov 01 06:26:55.032673 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.032639 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2491bf03-746jt"] Nov 01 06:26:55.035314 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.033281 4385 cpu_manager.go:414] "containerMap: removing container" podUID="753c422d-48cc-40ee-8673-208ed65fd311" containerName="istio-proxy" Nov 01 06:26:55.035314 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.033294 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="753c422d-48cc-40ee-8673-208ed65fd311" containerName="istio-proxy" Nov 01 06:26:55.035314 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.033326 4385 cpu_manager.go:414] "containerMap: removing container" podUID="12175cf8-da78-4493-8764-7d1f189e006d" containerName="revoke-keys" Nov 01 06:26:55.035314 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.033331 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12175cf8-da78-4493-8764-7d1f189e006d" containerName="revoke-keys" Nov 01 06:26:55.035314 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.033418 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="12175cf8-da78-4493-8764-7d1f189e006d" containerName="revoke-keys" Nov 01 06:26:55.035314 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.033429 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="753c422d-48cc-40ee-8673-208ed65fd311" containerName="istio-proxy" Nov 01 06:26:55.036802 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.036785 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2491bf03-746jt" Nov 01 06:26:55.094495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.094465 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2491bf03-746jt"] Nov 01 06:26:55.129616 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.129585 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmj7k\" (UniqueName: \"kubernetes.io/projected/09caa300-7be1-4c82-8a88-8b4abc2cba1c-kube-api-access-lmj7k\") pod \"maas-api-revoke-sub-2491bf03-746jt\" (UID: \"09caa300-7be1-4c82-8a88-8b4abc2cba1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2491bf03-746jt" Nov 01 06:26:55.129941 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.129919 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09caa300-7be1-4c82-8a88-8b4abc2cba1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2491bf03-746jt\" (UID: \"09caa300-7be1-4c82-8a88-8b4abc2cba1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2491bf03-746jt" Nov 01 06:26:55.130181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.130149 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09caa300-7be1-4c82-8a88-8b4abc2cba1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2491bf03-746jt\" (UID: \"09caa300-7be1-4c82-8a88-8b4abc2cba1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2491bf03-746jt" Nov 01 06:26:55.136405 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.136381 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:26:55.136607 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.136590 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\"" Nov 01 06:26:55.231492 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.231448 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09caa300-7be1-4c82-8a88-8b4abc2cba1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2491bf03-746jt\" (UID: \"09caa300-7be1-4c82-8a88-8b4abc2cba1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2491bf03-746jt" Nov 01 06:26:55.231705 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.231558 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09caa300-7be1-4c82-8a88-8b4abc2cba1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2491bf03-746jt\" (UID: \"09caa300-7be1-4c82-8a88-8b4abc2cba1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2491bf03-746jt" Nov 01 06:26:55.231705 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.231624 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmj7k\" (UniqueName: \"kubernetes.io/projected/09caa300-7be1-4c82-8a88-8b4abc2cba1c-kube-api-access-lmj7k\") pod \"maas-api-revoke-sub-2491bf03-746jt\" (UID: \"09caa300-7be1-4c82-8a88-8b4abc2cba1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2491bf03-746jt" Nov 01 06:26:55.232198 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.232172 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09caa300-7be1-4c82-8a88-8b4abc2cba1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2491bf03-746jt\" (UID: \"09caa300-7be1-4c82-8a88-8b4abc2cba1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2491bf03-746jt" Nov 01 06:26:55.234038 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.234010 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09caa300-7be1-4c82-8a88-8b4abc2cba1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2491bf03-746jt\" (UID: \"09caa300-7be1-4c82-8a88-8b4abc2cba1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2491bf03-746jt" Nov 01 06:26:55.290546 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.290476 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmj7k\" (UniqueName: \"kubernetes.io/projected/09caa300-7be1-4c82-8a88-8b4abc2cba1c-kube-api-access-lmj7k\") pod \"maas-api-revoke-sub-2491bf03-746jt\" (UID: \"09caa300-7be1-4c82-8a88-8b4abc2cba1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2491bf03-746jt" Nov 01 06:26:55.351789 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.351765 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2491bf03-746jt" Nov 01 06:26:55.731484 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.731453 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2491bf03-746jt"] Nov 01 06:26:55.731713 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:26:55.731688 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09caa300_7be1_4c82_8a88_8b4abc2cba1c.slice/crio-67ec08dbb4c6585b38e911a46863d44867556f8bd5396c8958ab1d80e1577fef WatchSource:0}: Error finding container 67ec08dbb4c6585b38e911a46863d44867556f8bd5396c8958ab1d80e1577fef: Status 404 returned error can't find the container with id 67ec08dbb4c6585b38e911a46863d44867556f8bd5396c8958ab1d80e1577fef Nov 01 06:26:55.923254 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:55.923216 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2491bf03-746jt" event={"ID":"09caa300-7be1-4c82-8a88-8b4abc2cba1c","Type":"ContainerStarted","Data":"67ec08dbb4c6585b38e911a46863d44867556f8bd5396c8958ab1d80e1577fef"} Nov 01 06:26:56.146286 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:56.146251 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:26:56.871881 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:56.871845 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:26:56.942900 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:56.942784 4385 generic.go:356] "Generic (PLEG): container finished" podID="09caa300-7be1-4c82-8a88-8b4abc2cba1c" containerID="5c596f1a9c8cba5128fa78bbb1350e98dd6dda819bdc14e07da1288d5a5347ce" exitCode=0 Nov 01 06:26:56.943077 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:56.943032 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2491bf03-746jt" event={"ID":"09caa300-7be1-4c82-8a88-8b4abc2cba1c","Type":"ContainerDied","Data":"5c596f1a9c8cba5128fa78bbb1350e98dd6dda819bdc14e07da1288d5a5347ce"} Nov 01 06:26:58.013936 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.013898 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4"] Nov 01 06:26:58.018630 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.018608 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" Nov 01 06:26:58.061198 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.061170 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhqbg\" (UniqueName: \"kubernetes.io/projected/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-kube-api-access-hhqbg\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4\" (UID: \"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" Nov 01 06:26:58.061397 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.061362 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4\" (UID: \"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" Nov 01 06:26:58.061508 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.061438 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4\" (UID: \"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" Nov 01 06:26:58.061554 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.061511 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4\" (UID: \"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" Nov 01 06:26:58.074453 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.074431 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4"] Nov 01 06:26:58.097886 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.097866 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2491bf03-746jt" Nov 01 06:26:58.117258 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.117237 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c79abe\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c79abe\"" Nov 01 06:26:58.117430 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.117408 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c79abe\"/\"kube-root-ca.crt\"" Nov 01 06:26:58.117497 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.117418 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c79abe\"/\"openshift-service-ca.crt\"" Nov 01 06:26:58.117497 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.117484 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c79abe\"/\"routing-overlay\"" Nov 01 06:26:58.120314 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.120294 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c79abe\"/\"agc-cred-1e9a97dd8714\"" Nov 01 06:26:58.120435 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.120384 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c79abe\"/\"payload-processing-external-model-e2e--a434a17c-dockercfg-thkg2\"" Nov 01 06:26:58.162864 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.162818 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09caa300-7be1-4c82-8a88-8b4abc2cba1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09caa300-7be1-4c82-8a88-8b4abc2cba1c-maas-api-openshift-ca\") pod \"09caa300-7be1-4c82-8a88-8b4abc2cba1c\" (UID: \"09caa300-7be1-4c82-8a88-8b4abc2cba1c\") " Nov 01 06:26:58.162978 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.162878 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09caa300-7be1-4c82-8a88-8b4abc2cba1c-kube-api-access-lmj7k\" (UniqueName: \"kubernetes.io/projected/09caa300-7be1-4c82-8a88-8b4abc2cba1c-kube-api-access-lmj7k\") pod \"09caa300-7be1-4c82-8a88-8b4abc2cba1c\" (UID: \"09caa300-7be1-4c82-8a88-8b4abc2cba1c\") " Nov 01 06:26:58.162978 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.162958 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09caa300-7be1-4c82-8a88-8b4abc2cba1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09caa300-7be1-4c82-8a88-8b4abc2cba1c-maas-api-xks-ca\") pod \"09caa300-7be1-4c82-8a88-8b4abc2cba1c\" (UID: \"09caa300-7be1-4c82-8a88-8b4abc2cba1c\") " Nov 01 06:26:58.163181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.163160 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09caa300-7be1-4c82-8a88-8b4abc2cba1c-maas-api-openshift-ca" pod "09caa300-7be1-4c82-8a88-8b4abc2cba1c" (UID: "09caa300-7be1-4c82-8a88-8b4abc2cba1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:26:58.163245 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.163221 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4\" (UID: \"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" Nov 01 06:26:58.163359 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.163341 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4\" (UID: \"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" Nov 01 06:26:58.163475 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.163408 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhqbg\" (UniqueName: \"kubernetes.io/projected/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-kube-api-access-hhqbg\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4\" (UID: \"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" Nov 01 06:26:58.163557 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.163531 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4\" (UID: \"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" Nov 01 06:26:58.163625 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.163612 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09caa300-7be1-4c82-8a88-8b4abc2cba1c-maas-api-openshift-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:58.163982 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.163959 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4\" (UID: \"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" Nov 01 06:26:58.164406 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.164381 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4\" (UID: \"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" Nov 01 06:26:58.165336 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.165314 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09caa300-7be1-4c82-8a88-8b4abc2cba1c-maas-api-xks-ca" pod "09caa300-7be1-4c82-8a88-8b4abc2cba1c" (UID: "09caa300-7be1-4c82-8a88-8b4abc2cba1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:26:58.165398 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.165367 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09caa300-7be1-4c82-8a88-8b4abc2cba1c-kube-api-access-lmj7k" pod "09caa300-7be1-4c82-8a88-8b4abc2cba1c" (UID: "09caa300-7be1-4c82-8a88-8b4abc2cba1c"). InnerVolumeSpecName "kube-api-access-lmj7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:26:58.166006 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.165989 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4\" (UID: \"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" Nov 01 06:26:58.220020 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.219956 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhqbg\" (UniqueName: \"kubernetes.io/projected/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-kube-api-access-hhqbg\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4\" (UID: \"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" Nov 01 06:26:58.264802 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.264779 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmj7k\" (UniqueName: \"kubernetes.io/projected/09caa300-7be1-4c82-8a88-8b4abc2cba1c-kube-api-access-lmj7k\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:58.264802 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.264802 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09caa300-7be1-4c82-8a88-8b4abc2cba1c-maas-api-xks-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:58.337158 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.337117 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" Nov 01 06:26:58.718195 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:26:58.718168 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ea9656_98d0_48eb_9ab7_e4cb44ca8c22.slice/crio-07b02148dcc1b9720366c038fa496d1d3b575279bfb44f2645c1fbeaa05ca1a6 WatchSource:0}: Error finding container 07b02148dcc1b9720366c038fa496d1d3b575279bfb44f2645c1fbeaa05ca1a6: Status 404 returned error can't find the container with id 07b02148dcc1b9720366c038fa496d1d3b575279bfb44f2645c1fbeaa05ca1a6 Nov 01 06:26:58.718345 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.718323 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4"] Nov 01 06:26:58.967345 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.967263 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2491bf03-746jt" event={"ID":"09caa300-7be1-4c82-8a88-8b4abc2cba1c","Type":"ContainerDied","Data":"67ec08dbb4c6585b38e911a46863d44867556f8bd5396c8958ab1d80e1577fef"} Nov 01 06:26:58.967345 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.967327 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2491bf03-746jt" Nov 01 06:26:58.967345 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.967343 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67ec08dbb4c6585b38e911a46863d44867556f8bd5396c8958ab1d80e1577fef" Nov 01 06:26:58.969654 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.969393 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" event={"ID":"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22","Type":"ContainerStarted","Data":"49e78743c3f5101e96a81b6e653a547ca747f00316c26b9114f1092bf22fb156"} Nov 01 06:26:58.969807 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.969630 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" Nov 01 06:26:58.969807 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:58.969696 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" event={"ID":"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22","Type":"ContainerStarted","Data":"07b02148dcc1b9720366c038fa496d1d3b575279bfb44f2645c1fbeaa05ca1a6"} Nov 01 06:26:59.083889 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.083824 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" podStartSLOduration=2.08381205 podStartE2EDuration="2.08381205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:26:59.081093548 +0000 UTC m=+1028.844893384" watchObservedRunningTime="2026-11-01 06:26:59.08381205 +0000 UTC m=+1028.847611907" Nov 01 06:26:59.266338 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.266259 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp"] Nov 01 06:26:59.266730 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.266717 4385 cpu_manager.go:414] "containerMap: removing container" podUID="09caa300-7be1-4c82-8a88-8b4abc2cba1c" containerName="revoke-keys" Nov 01 06:26:59.266771 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.266731 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09caa300-7be1-4c82-8a88-8b4abc2cba1c" containerName="revoke-keys" Nov 01 06:26:59.266857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.266844 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="09caa300-7be1-4c82-8a88-8b4abc2cba1c" containerName="revoke-keys" Nov 01 06:26:59.270307 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.270292 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" Nov 01 06:26:59.323325 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.323293 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"routing-overlay\"" Nov 01 06:26:59.323501 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.323374 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"payload-processing-external-model-plugins-e2e-extmod-b-dc6e48\"" Nov 01 06:26:59.323589 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.323505 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-2b38f786e9be\"" Nov 01 06:26:59.323589 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.323582 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-120370fe39d3\"" Nov 01 06:26:59.323674 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.323638 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"openshift-service-ca.crt\"" Nov 01 06:26:59.323706 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.323666 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"kube-root-ca.crt\"" Nov 01 06:26:59.323784 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.323768 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-a83cd424b5a5\"" Nov 01 06:26:59.326855 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.326809 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp"] Nov 01 06:26:59.357486 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.357461 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"payload-processing-external-model-e2e--958558de-dockercfg-qjzd6\"" Nov 01 06:26:59.376446 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.376422 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp\" (UID: \"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" Nov 01 06:26:59.376576 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.376490 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp78x\" (UniqueName: \"kubernetes.io/projected/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-kube-api-access-cp78x\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp\" (UID: \"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" Nov 01 06:26:59.376631 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.376597 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp\" (UID: \"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" Nov 01 06:26:59.376668 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.376641 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp\" (UID: \"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" Nov 01 06:26:59.478172 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.478132 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp\" (UID: \"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" Nov 01 06:26:59.478352 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.478186 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp\" (UID: \"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" Nov 01 06:26:59.478352 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.478261 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp\" (UID: \"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" Nov 01 06:26:59.478458 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.478360 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp78x\" (UniqueName: \"kubernetes.io/projected/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-kube-api-access-cp78x\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp\" (UID: \"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" Nov 01 06:26:59.479144 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.479116 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp\" (UID: \"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" Nov 01 06:26:59.479255 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.479170 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp\" (UID: \"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" Nov 01 06:26:59.480962 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.480943 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp\" (UID: \"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" Nov 01 06:26:59.534701 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.534629 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp78x\" (UniqueName: \"kubernetes.io/projected/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-kube-api-access-cp78x\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp\" (UID: \"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" Nov 01 06:26:59.584377 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.584342 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" Nov 01 06:26:59.963303 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.963276 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp"] Nov 01 06:26:59.963503 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:26:59.963484 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d1320e_82f2_4b5d_a0e3_9dd1ffe583a5.slice/crio-8254061a385365848f8615a1995a9b5cb1b036d18c929cdb81d8a86af42e5c7d WatchSource:0}: Error finding container 8254061a385365848f8615a1995a9b5cb1b036d18c929cdb81d8a86af42e5c7d: Status 404 returned error can't find the container with id 8254061a385365848f8615a1995a9b5cb1b036d18c929cdb81d8a86af42e5c7d Nov 01 06:26:59.978439 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:26:59.978409 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" event={"ID":"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5","Type":"ContainerStarted","Data":"8254061a385365848f8615a1995a9b5cb1b036d18c929cdb81d8a86af42e5c7d"} Nov 01 06:27:00.976015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:00.975981 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:00.986490 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:00.986451 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" event={"ID":"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5","Type":"ContainerStarted","Data":"9b750cf3091956ac83be4cf868a274c0d3dece0d669f82429965df6e03450111"} Nov 01 06:27:00.986639 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:00.986625 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" Nov 01 06:27:01.100235 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.100178 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" podStartSLOduration=2.100159712 podStartE2EDuration="2.100159712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:27:01.099453241 +0000 UTC m=+1030.863253099" watchObservedRunningTime="2026-11-01 06:27:01.100159712 +0000 UTC m=+1030.863959569" Nov 01 06:27:01.440375 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.440333 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" podUID="c6564d39-0b98-4ea2-9801-b31c8aa44ca3" containerName="main" containerID="cri-o://bb0c25b7bd7c62cb1829a34af4c8403e8e53fa27031e0a3515476a65e395a6e8" gracePeriod=45 Nov 01 06:27:01.704236 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.704175 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:27:01.801202 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.801171 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-model-cache\") pod \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " Nov 01 06:27:01.801202 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.801207 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-dshm\") pod \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " Nov 01 06:27:01.801463 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.801256 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-tls-certs\") pod \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " Nov 01 06:27:01.801525 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.801474 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-tmp-dir\") pod \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " Nov 01 06:27:01.801525 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.801502 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-home\" (UniqueName: \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-home\") pod \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " Nov 01 06:27:01.801626 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.801516 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-model-cache" pod "c6564d39-0b98-4ea2-9801-b31c8aa44ca3" (UID: "c6564d39-0b98-4ea2-9801-b31c8aa44ca3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:27:01.801626 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.801558 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-kube-api-access-bn2dz\" (UniqueName: \"kubernetes.io/projected/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-kube-api-access-bn2dz\") pod \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\" (UID: \"c6564d39-0b98-4ea2-9801-b31c8aa44ca3\") " Nov 01 06:27:01.801740 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.801707 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-tmp-dir" pod "c6564d39-0b98-4ea2-9801-b31c8aa44ca3" (UID: "c6564d39-0b98-4ea2-9801-b31c8aa44ca3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:27:01.801791 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.801775 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-home" pod "c6564d39-0b98-4ea2-9801-b31c8aa44ca3" (UID: "c6564d39-0b98-4ea2-9801-b31c8aa44ca3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:27:01.801981 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.801962 4385 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-tmp-dir\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:01.801981 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.801980 4385 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-home\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:01.802287 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.801989 4385 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-model-cache\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:01.803815 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.803791 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-dshm" pod "c6564d39-0b98-4ea2-9801-b31c8aa44ca3" (UID: "c6564d39-0b98-4ea2-9801-b31c8aa44ca3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:27:01.803815 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.803801 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-tls-certs" pod "c6564d39-0b98-4ea2-9801-b31c8aa44ca3" (UID: "c6564d39-0b98-4ea2-9801-b31c8aa44ca3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:27:01.803973 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.803799 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-kube-api-access-bn2dz" pod "c6564d39-0b98-4ea2-9801-b31c8aa44ca3" (UID: "c6564d39-0b98-4ea2-9801-b31c8aa44ca3"). InnerVolumeSpecName "kube-api-access-bn2dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:27:01.903085 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.903048 4385 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-tls-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:01.903085 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.903080 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn2dz\" (UniqueName: \"kubernetes.io/projected/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-kube-api-access-bn2dz\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:01.903284 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.903095 4385 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6564d39-0b98-4ea2-9801-b31c8aa44ca3-dshm\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:01.999813 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.999723 4385 generic.go:356] "Generic (PLEG): container finished" podID="c6564d39-0b98-4ea2-9801-b31c8aa44ca3" containerID="bb0c25b7bd7c62cb1829a34af4c8403e8e53fa27031e0a3515476a65e395a6e8" exitCode=0 Nov 01 06:27:01.999813 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.999770 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" Nov 01 06:27:01.999813 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.999805 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" event={"ID":"c6564d39-0b98-4ea2-9801-b31c8aa44ca3","Type":"ContainerDied","Data":"bb0c25b7bd7c62cb1829a34af4c8403e8e53fa27031e0a3515476a65e395a6e8"} Nov 01 06:27:02.000431 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.999862 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs" event={"ID":"c6564d39-0b98-4ea2-9801-b31c8aa44ca3","Type":"ContainerDied","Data":"34a063b1bcb4b4eb4da0e403b8ef64babcbecfbc5f705d5fee6464970da11d73"} Nov 01 06:27:02.000431 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:01.999883 4385 scope.go:122] "RemoveContainer" containerID="bb0c25b7bd7c62cb1829a34af4c8403e8e53fa27031e0a3515476a65e395a6e8" Nov 01 06:27:02.012407 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:02.012391 4385 scope.go:122] "RemoveContainer" containerID="bb0c25b7bd7c62cb1829a34af4c8403e8e53fa27031e0a3515476a65e395a6e8" Nov 01 06:27:02.012672 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:27:02.012655 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb0c25b7bd7c62cb1829a34af4c8403e8e53fa27031e0a3515476a65e395a6e8\": container with ID starting with bb0c25b7bd7c62cb1829a34af4c8403e8e53fa27031e0a3515476a65e395a6e8 not found: ID does not exist" containerID="bb0c25b7bd7c62cb1829a34af4c8403e8e53fa27031e0a3515476a65e395a6e8" Nov 01 06:27:02.012743 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:02.012694 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0c25b7bd7c62cb1829a34af4c8403e8e53fa27031e0a3515476a65e395a6e8"} err="failed to get container status \"bb0c25b7bd7c62cb1829a34af4c8403e8e53fa27031e0a3515476a65e395a6e8\": rpc error: code = NotFound desc = could not find container \"bb0c25b7bd7c62cb1829a34af4c8403e8e53fa27031e0a3515476a65e395a6e8\": container with ID starting with bb0c25b7bd7c62cb1829a34af4c8403e8e53fa27031e0a3515476a65e395a6e8 not found: ID does not exist" Nov 01 06:27:02.170021 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:02.169982 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs"] Nov 01 06:27:02.173526 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:02.173501 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-85584b/test-model-85584b-kserve-646c97f487-4cjzs"] Nov 01 06:27:02.385088 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:02.385056 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6564d39-0b98-4ea2-9801-b31c8aa44ca3" path="/var/lib/kubelet/pods/c6564d39-0b98-4ea2-9801-b31c8aa44ca3/volumes" Nov 01 06:27:05.910860 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:05.910476 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:06.858675 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:27:06.858598 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:27:08.113344 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:08.113311 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4"] Nov 01 06:27:08.116550 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:08.116525 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" Nov 01 06:27:08.264814 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:08.264753 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:09.249169 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:09.249130 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:09.699825 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:09.699791 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:10.731938 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:10.731903 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:12.002891 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.002861 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" Nov 01 06:27:12.288330 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.288239 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg"] Nov 01 06:27:12.288893 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.288874 4385 cpu_manager.go:414] "containerMap: removing container" podUID="c6564d39-0b98-4ea2-9801-b31c8aa44ca3" containerName="main" Nov 01 06:27:12.288893 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.288894 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6564d39-0b98-4ea2-9801-b31c8aa44ca3" containerName="main" Nov 01 06:27:12.289079 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.289049 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6564d39-0b98-4ea2-9801-b31c8aa44ca3" containerName="main" Nov 01 06:27:12.292322 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.292304 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.387162 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.387131 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg"] Nov 01 06:27:12.481133 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.481095 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.481283 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.481156 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.481283 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.481224 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qhzr\" (UniqueName: \"kubernetes.io/projected/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-kube-api-access-7qhzr\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.481419 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.481296 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.481419 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.481409 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.481535 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.481447 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.481715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.481681 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.481811 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.481777 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.481907 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.481856 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.582845 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.582807 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.583012 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.582882 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.583012 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.582923 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qhzr\" (UniqueName: \"kubernetes.io/projected/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-kube-api-access-7qhzr\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.583012 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.582947 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.583012 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.582973 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.583012 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.583001 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.583270 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.583148 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.583270 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.583215 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.583270 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.583257 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.583553 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.583526 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.583675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.583565 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.583675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.583627 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.583675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.583653 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.583787 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.583755 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.585867 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.585846 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.586098 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.586077 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.639336 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.639314 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.639451 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.639387 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qhzr\" (UniqueName: \"kubernetes.io/projected/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-kube-api-access-7qhzr\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:12.904970 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:12.904896 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:13.114647 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:13.114608 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" podUID="39ea9656-98d0-48eb-9ab7-e4cb44ca8c22" containerName="payload-processing" containerID="cri-o://49e78743c3f5101e96a81b6e653a547ca747f00316c26b9114f1092bf22fb156" gracePeriod=25 Nov 01 06:27:13.330658 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:27:13.330617 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdf0b3a_718c_49d1_9dd9_bd90b5a44e63.slice/crio-547cd809868b11965b12b24588f42712bab7955361e3604f0d68036e40f3adb4 WatchSource:0}: Error finding container 547cd809868b11965b12b24588f42712bab7955361e3604f0d68036e40f3adb4: Status 404 returned error can't find the container with id 547cd809868b11965b12b24588f42712bab7955361e3604f0d68036e40f3adb4 Nov 01 06:27:13.330819 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:13.330698 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg"] Nov 01 06:27:13.333816 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:13.333779 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:27:13.333961 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:13.333885 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:27:13.333961 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:13.333929 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:27:13.357227 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:13.357209 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" Nov 01 06:27:13.494359 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:13.494266 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-routing-overlay-volume\") pod \"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22\" (UID: \"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22\") " Nov 01 06:27:13.494527 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:13.494485 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-provider-credentials\" (UniqueName: \"kubernetes.io/projected/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-provider-credentials\") pod \"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22\" (UID: \"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22\") " Nov 01 06:27:13.494527 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:13.494522 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-kube-api-access-hhqbg\" (UniqueName: \"kubernetes.io/projected/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-kube-api-access-hhqbg\") pod \"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22\" (UID: \"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22\") " Nov 01 06:27:13.494652 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:13.494557 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-plugins-config-volume\") pod \"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22\" (UID: \"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22\") " Nov 01 06:27:13.495509 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:13.495382 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-plugins-config-volume" pod "39ea9656-98d0-48eb-9ab7-e4cb44ca8c22" (UID: "39ea9656-98d0-48eb-9ab7-e4cb44ca8c22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:27:13.495509 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:13.495438 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-routing-overlay-volume" pod "39ea9656-98d0-48eb-9ab7-e4cb44ca8c22" (UID: "39ea9656-98d0-48eb-9ab7-e4cb44ca8c22"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:27:13.498307 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:13.498280 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-kube-api-access-hhqbg" pod "39ea9656-98d0-48eb-9ab7-e4cb44ca8c22" (UID: "39ea9656-98d0-48eb-9ab7-e4cb44ca8c22"). InnerVolumeSpecName "kube-api-access-hhqbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:27:13.498414 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:13.498374 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-provider-credentials" pod "39ea9656-98d0-48eb-9ab7-e4cb44ca8c22" (UID: "39ea9656-98d0-48eb-9ab7-e4cb44ca8c22"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:27:13.596248 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:13.596216 4385 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-provider-credentials\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:13.596248 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:13.596247 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhqbg\" (UniqueName: \"kubernetes.io/projected/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-kube-api-access-hhqbg\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:13.596446 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:13.596263 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:13.596446 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:13.596276 4385 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22-routing-overlay-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:14.103552 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:14.103514 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" event={"ID":"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63","Type":"ContainerStarted","Data":"321811122f3e95c3fe018aac6fa2b26a61fc2487d810a669b1685f900e12eb90"} Nov 01 06:27:14.103904 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:14.103565 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" event={"ID":"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63","Type":"ContainerStarted","Data":"547cd809868b11965b12b24588f42712bab7955361e3604f0d68036e40f3adb4"} Nov 01 06:27:14.105106 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:14.105079 4385 generic.go:356] "Generic (PLEG): container finished" podID="39ea9656-98d0-48eb-9ab7-e4cb44ca8c22" containerID="49e78743c3f5101e96a81b6e653a547ca747f00316c26b9114f1092bf22fb156" exitCode=0 Nov 01 06:27:14.105231 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:14.105110 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" event={"ID":"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22","Type":"ContainerDied","Data":"49e78743c3f5101e96a81b6e653a547ca747f00316c26b9114f1092bf22fb156"} Nov 01 06:27:14.105231 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:14.105133 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" Nov 01 06:27:14.105231 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:14.105165 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4" event={"ID":"39ea9656-98d0-48eb-9ab7-e4cb44ca8c22","Type":"ContainerDied","Data":"07b02148dcc1b9720366c038fa496d1d3b575279bfb44f2645c1fbeaa05ca1a6"} Nov 01 06:27:14.105231 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:14.105191 4385 scope.go:122] "RemoveContainer" containerID="49e78743c3f5101e96a81b6e653a547ca747f00316c26b9114f1092bf22fb156" Nov 01 06:27:14.115712 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:14.115694 4385 scope.go:122] "RemoveContainer" containerID="49e78743c3f5101e96a81b6e653a547ca747f00316c26b9114f1092bf22fb156" Nov 01 06:27:14.116009 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:27:14.115962 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49e78743c3f5101e96a81b6e653a547ca747f00316c26b9114f1092bf22fb156\": container with ID starting with 49e78743c3f5101e96a81b6e653a547ca747f00316c26b9114f1092bf22fb156 not found: ID does not exist" containerID="49e78743c3f5101e96a81b6e653a547ca747f00316c26b9114f1092bf22fb156" Nov 01 06:27:14.116009 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:14.115983 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49e78743c3f5101e96a81b6e653a547ca747f00316c26b9114f1092bf22fb156"} err="failed to get container status \"49e78743c3f5101e96a81b6e653a547ca747f00316c26b9114f1092bf22fb156\": rpc error: code = NotFound desc = could not find container \"49e78743c3f5101e96a81b6e653a547ca747f00316c26b9114f1092bf22fb156\": container with ID starting with 49e78743c3f5101e96a81b6e653a547ca747f00316c26b9114f1092bf22fb156 not found: ID does not exist" Nov 01 06:27:14.274859 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:14.271714 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" podStartSLOduration=2.271694868 podStartE2EDuration="2.271694868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:27:14.268001813 +0000 UTC m=+1044.031801667" watchObservedRunningTime="2026-11-01 06:27:14.271694868 +0000 UTC m=+1044.035494726" Nov 01 06:27:14.406326 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:14.406242 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4"] Nov 01 06:27:14.409076 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:14.409055 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-569ddq9h4"] Nov 01 06:27:14.906071 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:14.906035 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:14.907279 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:14.907249 4385 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 06:27:14.907407 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:14.907308 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" podUID="4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 06:27:15.592859 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:15.592759 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:15.739227 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:15.739191 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:15.905886 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:15.905793 4385 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 06:27:15.905886 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:15.905855 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" podUID="4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 06:27:16.402589 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:16.402553 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ea9656-98d0-48eb-9ab7-e4cb44ca8c22" path="/var/lib/kubelet/pods/39ea9656-98d0-48eb-9ab7-e4cb44ca8c22/volumes" Nov 01 06:27:16.905498 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:16.905464 4385 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 06:27:16.905967 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:16.905515 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" podUID="4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 06:27:17.603134 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:17.603095 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466"] Nov 01 06:27:17.709201 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:17.709164 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx"] Nov 01 06:27:17.906058 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:17.905960 4385 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 06:27:17.906618 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:17.906586 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" podUID="4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 06:27:17.955489 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:17.955427 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:18.100030 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:18.099983 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:18.906196 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:18.906156 4385 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 01 06:27:18.906593 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:18.906221 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" podUID="4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 01 06:27:19.030899 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:19.030858 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:19.279395 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:19.279275 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:20.099140 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:20.099101 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:20.107112 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:20.107084 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:20.107885 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:20.107854 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:20.108932 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:20.108914 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:27:20.322917 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:20.322874 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:20.482034 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:20.481948 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:20.738789 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:20.738688 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:20.884844 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:20.884785 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:21.544891 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:21.544820 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:22.526275 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.526241 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69"] Nov 01 06:27:22.526754 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.526740 4385 cpu_manager.go:414] "containerMap: removing container" podUID="39ea9656-98d0-48eb-9ab7-e4cb44ca8c22" containerName="payload-processing" Nov 01 06:27:22.526798 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.526756 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ea9656-98d0-48eb-9ab7-e4cb44ca8c22" containerName="payload-processing" Nov 01 06:27:22.526868 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.526858 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ea9656-98d0-48eb-9ab7-e4cb44ca8c22" containerName="payload-processing" Nov 01 06:27:22.532385 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.532345 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:22.588318 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.588279 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69"] Nov 01 06:27:22.603913 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.603829 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466" podUID="48ea0a3f-5ef6-45fe-ba2a-894a232e81b4" containerName="payload-processing" containerID="cri-o://0d81d8ae9a97572d6a10094e28af5a4ef7ef31ee7c2640f1032a103695ec49de" gracePeriod=25 Nov 01 06:27:22.631726 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.631693 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8ff537\"/\"distinct-w2-8ff537-kserve-self-signed-certs\"" Nov 01 06:27:22.633206 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.633183 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v4lf\" (UniqueName: \"kubernetes.io/projected/695583a3-4ef8-4cad-ba76-69791506f2f3-kube-api-access-4v4lf\") pod \"distinct-w2-8ff537-kserve-7b5d9457d-hsf69\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:22.633283 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.633221 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-home\") pod \"distinct-w2-8ff537-kserve-7b5d9457d-hsf69\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:22.633355 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.633324 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/695583a3-4ef8-4cad-ba76-69791506f2f3-tls-certs\") pod \"distinct-w2-8ff537-kserve-7b5d9457d-hsf69\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:22.633405 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.633379 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-dshm\") pod \"distinct-w2-8ff537-kserve-7b5d9457d-hsf69\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:22.633448 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.633437 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-tmp-dir\") pod \"distinct-w2-8ff537-kserve-7b5d9457d-hsf69\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:22.633572 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.633556 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-model-cache\") pod \"distinct-w2-8ff537-kserve-7b5d9457d-hsf69\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:22.636365 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.636332 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:22.709649 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.709602 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx" podUID="0213cfe7-3498-4570-8de5-1bb46d87503c" containerName="payload-pre-processing" containerID="cri-o://11a3256fac39f47cd3fc52fa388267e2cb70cf02219fb565ecd5648c2cfcbf2a" gracePeriod=25 Nov 01 06:27:22.735032 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.734994 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v4lf\" (UniqueName: \"kubernetes.io/projected/695583a3-4ef8-4cad-ba76-69791506f2f3-kube-api-access-4v4lf\") pod \"distinct-w2-8ff537-kserve-7b5d9457d-hsf69\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:22.735283 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.735259 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-home\") pod \"distinct-w2-8ff537-kserve-7b5d9457d-hsf69\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:22.735418 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.735314 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/695583a3-4ef8-4cad-ba76-69791506f2f3-tls-certs\") pod \"distinct-w2-8ff537-kserve-7b5d9457d-hsf69\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:22.735418 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.735350 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-dshm\") pod \"distinct-w2-8ff537-kserve-7b5d9457d-hsf69\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:22.735418 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.735395 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-tmp-dir\") pod \"distinct-w2-8ff537-kserve-7b5d9457d-hsf69\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:22.735550 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.735453 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-model-cache\") pod \"distinct-w2-8ff537-kserve-7b5d9457d-hsf69\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:22.735787 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.735764 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-home\") pod \"distinct-w2-8ff537-kserve-7b5d9457d-hsf69\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:22.735895 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.735828 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-model-cache\") pod \"distinct-w2-8ff537-kserve-7b5d9457d-hsf69\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:22.736148 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.736117 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-tmp-dir\") pod \"distinct-w2-8ff537-kserve-7b5d9457d-hsf69\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:22.778911 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.756473 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/695583a3-4ef8-4cad-ba76-69791506f2f3-tls-certs\") pod \"distinct-w2-8ff537-kserve-7b5d9457d-hsf69\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:22.789860 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.783172 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-dshm\") pod \"distinct-w2-8ff537-kserve-7b5d9457d-hsf69\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:22.808897 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.807522 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v4lf\" (UniqueName: \"kubernetes.io/projected/695583a3-4ef8-4cad-ba76-69791506f2f3-kube-api-access-4v4lf\") pod \"distinct-w2-8ff537-kserve-7b5d9457d-hsf69\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:22.848447 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:22.848412 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:23.323485 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:23.323368 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69"] Nov 01 06:27:24.190564 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:24.190521 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" event={"ID":"695583a3-4ef8-4cad-ba76-69791506f2f3","Type":"ContainerStarted","Data":"836e522c433484c09f0b99c6f55c13812bdfa881d3014e60feb1490f8facd7a7"} Nov 01 06:27:24.191023 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:24.190578 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" event={"ID":"695583a3-4ef8-4cad-ba76-69791506f2f3","Type":"ContainerStarted","Data":"8a988f1ad9f22f1346b03cfd3146d01b94a54b010e1437fa096c8caa2c38a22c"} Nov 01 06:27:24.307865 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:24.307786 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" podStartSLOduration=2.307766891 podStartE2EDuration="2.307766891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:27:24.304244485 +0000 UTC m=+1054.068044344" watchObservedRunningTime="2026-11-01 06:27:24.307766891 +0000 UTC m=+1054.071566752" Nov 01 06:27:24.849252 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:24.849222 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:25.405626 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:25.405577 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:25.618420 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:25.618384 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:25.845269 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:25.845239 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:26.008370 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:26.008331 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:28.602522 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:28.602485 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:28.774870 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:28.774801 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:30.914345 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:30.914311 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:31.425966 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:31.425926 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466" podUID="48ea0a3f-5ef6-45fe-ba2a-894a232e81b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:27:31.426150 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:31.425965 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx" podUID="0213cfe7-3498-4570-8de5-1bb46d87503c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:27:32.849317 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:32.849283 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:32.862969 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:32.862937 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:33.271373 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:33.271264 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 06:27:35.724730 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:35.724700 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:36.789656 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:27:36.789618 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:27:39.918928 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:39.918881 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:40.427958 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:40.427908 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:41.426700 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:41.426668 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466" podUID="48ea0a3f-5ef6-45fe-ba2a-894a232e81b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:27:41.427134 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:41.426733 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx" podUID="0213cfe7-3498-4570-8de5-1bb46d87503c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:27:42.901345 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:42.901314 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466" Nov 01 06:27:42.979369 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:42.979346 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx" Nov 01 06:27:43.000191 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.000163 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0213cfe7-3498-4570-8de5-1bb46d87503c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0213cfe7-3498-4570-8de5-1bb46d87503c-plugins-config-volume\") pod \"0213cfe7-3498-4570-8de5-1bb46d87503c\" (UID: \"0213cfe7-3498-4570-8de5-1bb46d87503c\") " Nov 01 06:27:43.000332 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.000298 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48ea0a3f-5ef6-45fe-ba2a-894a232e81b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ea0a3f-5ef6-45fe-ba2a-894a232e81b4-plugins-config-volume\") pod \"48ea0a3f-5ef6-45fe-ba2a-894a232e81b4\" (UID: \"48ea0a3f-5ef6-45fe-ba2a-894a232e81b4\") " Nov 01 06:27:43.000385 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.000368 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0213cfe7-3498-4570-8de5-1bb46d87503c-kube-api-access-2hxvx\" (UniqueName: \"kubernetes.io/projected/0213cfe7-3498-4570-8de5-1bb46d87503c-kube-api-access-2hxvx\") pod \"0213cfe7-3498-4570-8de5-1bb46d87503c\" (UID: \"0213cfe7-3498-4570-8de5-1bb46d87503c\") " Nov 01 06:27:43.000442 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.000426 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48ea0a3f-5ef6-45fe-ba2a-894a232e81b4-kube-api-access-ftx2f\" (UniqueName: \"kubernetes.io/projected/48ea0a3f-5ef6-45fe-ba2a-894a232e81b4-kube-api-access-ftx2f\") pod \"48ea0a3f-5ef6-45fe-ba2a-894a232e81b4\" (UID: \"48ea0a3f-5ef6-45fe-ba2a-894a232e81b4\") " Nov 01 06:27:43.000595 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.000577 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0213cfe7-3498-4570-8de5-1bb46d87503c-plugins-config-volume" pod "0213cfe7-3498-4570-8de5-1bb46d87503c" (UID: "0213cfe7-3498-4570-8de5-1bb46d87503c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:27:43.000658 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.000635 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ea0a3f-5ef6-45fe-ba2a-894a232e81b4-plugins-config-volume" pod "48ea0a3f-5ef6-45fe-ba2a-894a232e81b4" (UID: "48ea0a3f-5ef6-45fe-ba2a-894a232e81b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:27:43.000882 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.000828 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0213cfe7-3498-4570-8de5-1bb46d87503c-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:43.000882 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.000877 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ea0a3f-5ef6-45fe-ba2a-894a232e81b4-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:43.002627 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.002605 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0213cfe7-3498-4570-8de5-1bb46d87503c-kube-api-access-2hxvx" pod "0213cfe7-3498-4570-8de5-1bb46d87503c" (UID: "0213cfe7-3498-4570-8de5-1bb46d87503c"). InnerVolumeSpecName "kube-api-access-2hxvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:27:43.002728 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.002625 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ea0a3f-5ef6-45fe-ba2a-894a232e81b4-kube-api-access-ftx2f" pod "48ea0a3f-5ef6-45fe-ba2a-894a232e81b4" (UID: "48ea0a3f-5ef6-45fe-ba2a-894a232e81b4"). InnerVolumeSpecName "kube-api-access-ftx2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:27:43.101944 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.101909 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hxvx\" (UniqueName: \"kubernetes.io/projected/0213cfe7-3498-4570-8de5-1bb46d87503c-kube-api-access-2hxvx\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:43.101944 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.101938 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftx2f\" (UniqueName: \"kubernetes.io/projected/48ea0a3f-5ef6-45fe-ba2a-894a232e81b4-kube-api-access-ftx2f\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:43.338717 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.338688 4385 generic.go:356] "Generic (PLEG): container finished" podID="0213cfe7-3498-4570-8de5-1bb46d87503c" containerID="11a3256fac39f47cd3fc52fa388267e2cb70cf02219fb565ecd5648c2cfcbf2a" exitCode=0 Nov 01 06:27:43.339011 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.338749 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx" Nov 01 06:27:43.339011 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.338777 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx" event={"ID":"0213cfe7-3498-4570-8de5-1bb46d87503c","Type":"ContainerDied","Data":"11a3256fac39f47cd3fc52fa388267e2cb70cf02219fb565ecd5648c2cfcbf2a"} Nov 01 06:27:43.339011 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.338869 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx" event={"ID":"0213cfe7-3498-4570-8de5-1bb46d87503c","Type":"ContainerDied","Data":"f8eeb294360f430df44c5837cf0f4701b8e8e16be5b3e856daa968cfeb414dba"} Nov 01 06:27:43.339011 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.338898 4385 scope.go:122] "RemoveContainer" containerID="11a3256fac39f47cd3fc52fa388267e2cb70cf02219fb565ecd5648c2cfcbf2a" Nov 01 06:27:43.340270 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.340242 4385 generic.go:356] "Generic (PLEG): container finished" podID="48ea0a3f-5ef6-45fe-ba2a-894a232e81b4" containerID="0d81d8ae9a97572d6a10094e28af5a4ef7ef31ee7c2640f1032a103695ec49de" exitCode=0 Nov 01 06:27:43.340379 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.340280 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466" event={"ID":"48ea0a3f-5ef6-45fe-ba2a-894a232e81b4","Type":"ContainerDied","Data":"0d81d8ae9a97572d6a10094e28af5a4ef7ef31ee7c2640f1032a103695ec49de"} Nov 01 06:27:43.340379 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.340300 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466" Nov 01 06:27:43.340379 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.340334 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466" event={"ID":"48ea0a3f-5ef6-45fe-ba2a-894a232e81b4","Type":"ContainerDied","Data":"35e4319eae2c297d03dfc678345a86c6a97092928c407c9112ab0ee839729d86"} Nov 01 06:27:43.349007 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.348939 4385 scope.go:122] "RemoveContainer" containerID="11a3256fac39f47cd3fc52fa388267e2cb70cf02219fb565ecd5648c2cfcbf2a" Nov 01 06:27:43.349219 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:27:43.349201 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a3256fac39f47cd3fc52fa388267e2cb70cf02219fb565ecd5648c2cfcbf2a\": container with ID starting with 11a3256fac39f47cd3fc52fa388267e2cb70cf02219fb565ecd5648c2cfcbf2a not found: ID does not exist" containerID="11a3256fac39f47cd3fc52fa388267e2cb70cf02219fb565ecd5648c2cfcbf2a" Nov 01 06:27:43.349274 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.349227 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a3256fac39f47cd3fc52fa388267e2cb70cf02219fb565ecd5648c2cfcbf2a"} err="failed to get container status \"11a3256fac39f47cd3fc52fa388267e2cb70cf02219fb565ecd5648c2cfcbf2a\": rpc error: code = NotFound desc = could not find container \"11a3256fac39f47cd3fc52fa388267e2cb70cf02219fb565ecd5648c2cfcbf2a\": container with ID starting with 11a3256fac39f47cd3fc52fa388267e2cb70cf02219fb565ecd5648c2cfcbf2a not found: ID does not exist" Nov 01 06:27:43.349274 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.349244 4385 scope.go:122] "RemoveContainer" containerID="0d81d8ae9a97572d6a10094e28af5a4ef7ef31ee7c2640f1032a103695ec49de" Nov 01 06:27:43.358862 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.358823 4385 scope.go:122] "RemoveContainer" containerID="0d81d8ae9a97572d6a10094e28af5a4ef7ef31ee7c2640f1032a103695ec49de" Nov 01 06:27:43.359149 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:27:43.359131 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d81d8ae9a97572d6a10094e28af5a4ef7ef31ee7c2640f1032a103695ec49de\": container with ID starting with 0d81d8ae9a97572d6a10094e28af5a4ef7ef31ee7c2640f1032a103695ec49de not found: ID does not exist" containerID="0d81d8ae9a97572d6a10094e28af5a4ef7ef31ee7c2640f1032a103695ec49de" Nov 01 06:27:43.359211 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.359160 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d81d8ae9a97572d6a10094e28af5a4ef7ef31ee7c2640f1032a103695ec49de"} err="failed to get container status \"0d81d8ae9a97572d6a10094e28af5a4ef7ef31ee7c2640f1032a103695ec49de\": rpc error: code = NotFound desc = could not find container \"0d81d8ae9a97572d6a10094e28af5a4ef7ef31ee7c2640f1032a103695ec49de\": container with ID starting with 0d81d8ae9a97572d6a10094e28af5a4ef7ef31ee7c2640f1032a103695ec49de not found: ID does not exist" Nov 01 06:27:43.503851 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.503797 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466"] Nov 01 06:27:43.507450 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.507427 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-85584b-6f56f6ff49-ls466"] Nov 01 06:27:43.616293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.616225 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx"] Nov 01 06:27:43.619869 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:43.619822 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-85584b-5ccf7b6dff-d8ldx"] Nov 01 06:27:44.431859 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:44.428628 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0213cfe7-3498-4570-8de5-1bb46d87503c" path="/var/lib/kubelet/pods/0213cfe7-3498-4570-8de5-1bb46d87503c/volumes" Nov 01 06:27:44.443924 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:44.442237 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48ea0a3f-5ef6-45fe-ba2a-894a232e81b4" path="/var/lib/kubelet/pods/48ea0a3f-5ef6-45fe-ba2a-894a232e81b4/volumes" Nov 01 06:27:45.609853 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:45.609805 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:49.682491 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:49.682456 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:51.591527 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:51.591497 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:52.842487 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:52.842442 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:52.989898 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:52.989865 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:54.338876 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:54.338819 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:54.846415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:54.846385 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:55.265199 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:55.265117 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:55.603648 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:55.603613 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:55.810648 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:55.810615 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:56.620645 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.620615 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj"] Nov 01 06:27:56.621171 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.621155 4385 cpu_manager.go:414] "containerMap: removing container" podUID="48ea0a3f-5ef6-45fe-ba2a-894a232e81b4" containerName="payload-processing" Nov 01 06:27:56.621223 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.621173 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ea0a3f-5ef6-45fe-ba2a-894a232e81b4" containerName="payload-processing" Nov 01 06:27:56.621223 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.621189 4385 cpu_manager.go:414] "containerMap: removing container" podUID="0213cfe7-3498-4570-8de5-1bb46d87503c" containerName="payload-pre-processing" Nov 01 06:27:56.621223 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.621195 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0213cfe7-3498-4570-8de5-1bb46d87503c" containerName="payload-pre-processing" Nov 01 06:27:56.621322 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.621291 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ea0a3f-5ef6-45fe-ba2a-894a232e81b4" containerName="payload-processing" Nov 01 06:27:56.621322 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.621298 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="0213cfe7-3498-4570-8de5-1bb46d87503c" containerName="payload-pre-processing" Nov 01 06:27:56.626180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.626163 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:27:56.682675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.682633 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-tmp-dir\") pod \"distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:27:56.682996 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.682972 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q96zb\" (UniqueName: \"kubernetes.io/projected/4001c2de-286a-4407-9c74-0cc2492bc8ca-kube-api-access-q96zb\") pod \"distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:27:56.686855 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.683215 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-dshm\") pod \"distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:27:56.692971 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.692945 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4001c2de-286a-4407-9c74-0cc2492bc8ca-tls-certs\") pod \"distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:27:56.693176 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.693150 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-home\") pod \"distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:27:56.693308 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.693185 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-model-cache\") pod \"distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:27:56.699655 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.699601 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj"] Nov 01 06:27:56.721536 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.721458 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:27:56.730033 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.729505 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8ff537\"/\"distinct-2-w2-8ff537-kserve-self-signed-certs\"" Nov 01 06:27:56.805651 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.805613 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4001c2de-286a-4407-9c74-0cc2492bc8ca-tls-certs\") pod \"distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:27:56.805878 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.805685 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-home\") pod \"distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:27:56.805878 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.805709 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-model-cache\") pod \"distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:27:56.805878 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.805765 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-tmp-dir\") pod \"distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:27:56.805878 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.805810 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q96zb\" (UniqueName: \"kubernetes.io/projected/4001c2de-286a-4407-9c74-0cc2492bc8ca-kube-api-access-q96zb\") pod \"distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:27:56.806170 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.805936 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-dshm\") pod \"distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:27:56.806569 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.806538 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-tmp-dir\") pod \"distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:27:56.806676 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.806582 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-model-cache\") pod \"distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:27:56.806676 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.806630 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-home\") pod \"distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:27:56.809010 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.808981 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-dshm\") pod \"distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:27:56.809408 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.809381 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4001c2de-286a-4407-9c74-0cc2492bc8ca-tls-certs\") pod \"distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:27:56.871523 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.871420 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q96zb\" (UniqueName: \"kubernetes.io/projected/4001c2de-286a-4407-9c74-0cc2492bc8ca-kube-api-access-q96zb\") pod \"distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:27:56.950452 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:56.950133 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:27:57.452173 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:57.452050 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj"] Nov 01 06:27:57.458895 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:27:57.458864 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4001c2de_286a_4407_9c74_0cc2492bc8ca.slice/crio-9d90ff53a1e74d6f7eda06a766959d010fe94c0026da6e447d156e77ed35d841 WatchSource:0}: Error finding container 9d90ff53a1e74d6f7eda06a766959d010fe94c0026da6e447d156e77ed35d841: Status 404 returned error can't find the container with id 9d90ff53a1e74d6f7eda06a766959d010fe94c0026da6e447d156e77ed35d841 Nov 01 06:27:58.472028 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:58.471989 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" event={"ID":"4001c2de-286a-4407-9c74-0cc2492bc8ca","Type":"ContainerStarted","Data":"43fcc0d977745f31f2c8a1edd5b8853d912771b841c0808bc59e2f0d335ba79b"} Nov 01 06:27:58.472523 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:58.472043 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" event={"ID":"4001c2de-286a-4407-9c74-0cc2492bc8ca","Type":"ContainerStarted","Data":"9d90ff53a1e74d6f7eda06a766959d010fe94c0026da6e447d156e77ed35d841"} Nov 01 06:27:58.584923 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:58.584871 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" podStartSLOduration=2.584853147 podStartE2EDuration="2.584853147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:27:58.583743515 +0000 UTC m=+1088.347543373" watchObservedRunningTime="2026-11-01 06:27:58.584853147 +0000 UTC m=+1088.348653007" Nov 01 06:27:58.954010 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:58.953977 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:27:59.074450 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:27:59.074412 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:01.434901 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:01.434865 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:02.140102 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:02.140057 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx"] Nov 01 06:28:02.514625 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:02.514557 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:02.662369 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:02.662327 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:02.843405 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:02.843371 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:03.237199 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:03.237108 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:03.509367 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:03.508382 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:04.285690 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:04.285659 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:06.781317 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:28:06.781283 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:28:06.954034 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:06.953999 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:28:06.966763 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:06.966739 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:28:07.141286 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:07.141244 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx" podUID="7731ee91-483d-4396-8146-68e18afdd3c4" containerName="payload-processing" containerID="cri-o://53686ef8a8bd30cc543fd88da987803e64ea51365fff209b4de681a7e29d4139" gracePeriod=25 Nov 01 06:28:07.429211 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:07.429117 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:07.551159 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:07.551133 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 06:28:07.787045 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:07.786967 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx" podUID="7731ee91-483d-4396-8146-68e18afdd3c4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:28:09.415171 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:09.415134 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:09.573853 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:09.573800 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:09.877927 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:09.877890 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:13.719860 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:13.695387 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:16.169094 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:16.169062 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:17.786428 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:17.786389 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx" podUID="7731ee91-483d-4396-8146-68e18afdd3c4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:28:19.817933 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:19.817902 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c96ff431-l79th"] Nov 01 06:28:19.824857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:19.824167 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96ff431-l79th" Nov 01 06:28:19.878101 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:19.878073 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c96ff431-l79th"] Nov 01 06:28:19.921289 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:19.921252 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beeb8104-b759-478c-8529-f83a49b5c541-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c96ff431-l79th\" (UID: \"beeb8104-b759-478c-8529-f83a49b5c541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96ff431-l79th" Nov 01 06:28:19.921472 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:19.921365 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beeb8104-b759-478c-8529-f83a49b5c541-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c96ff431-l79th\" (UID: \"beeb8104-b759-478c-8529-f83a49b5c541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96ff431-l79th" Nov 01 06:28:19.921472 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:19.921432 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf5fb\" (UniqueName: \"kubernetes.io/projected/beeb8104-b759-478c-8529-f83a49b5c541-kube-api-access-bf5fb\") pod \"maas-api-revoke-sub-c96ff431-l79th\" (UID: \"beeb8104-b759-478c-8529-f83a49b5c541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96ff431-l79th" Nov 01 06:28:19.922378 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:19.922357 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:28:19.922488 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:19.922412 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\"" Nov 01 06:28:20.022697 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:20.022664 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf5fb\" (UniqueName: \"kubernetes.io/projected/beeb8104-b759-478c-8529-f83a49b5c541-kube-api-access-bf5fb\") pod \"maas-api-revoke-sub-c96ff431-l79th\" (UID: \"beeb8104-b759-478c-8529-f83a49b5c541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96ff431-l79th" Nov 01 06:28:20.022911 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:20.022768 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beeb8104-b759-478c-8529-f83a49b5c541-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c96ff431-l79th\" (UID: \"beeb8104-b759-478c-8529-f83a49b5c541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96ff431-l79th" Nov 01 06:28:20.022911 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:20.022882 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beeb8104-b759-478c-8529-f83a49b5c541-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c96ff431-l79th\" (UID: \"beeb8104-b759-478c-8529-f83a49b5c541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96ff431-l79th" Nov 01 06:28:20.023549 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:20.023522 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beeb8104-b759-478c-8529-f83a49b5c541-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c96ff431-l79th\" (UID: \"beeb8104-b759-478c-8529-f83a49b5c541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96ff431-l79th" Nov 01 06:28:20.025459 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:20.025438 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beeb8104-b759-478c-8529-f83a49b5c541-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c96ff431-l79th\" (UID: \"beeb8104-b759-478c-8529-f83a49b5c541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96ff431-l79th" Nov 01 06:28:20.080084 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:20.080057 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf5fb\" (UniqueName: \"kubernetes.io/projected/beeb8104-b759-478c-8529-f83a49b5c541-kube-api-access-bf5fb\") pod \"maas-api-revoke-sub-c96ff431-l79th\" (UID: \"beeb8104-b759-478c-8529-f83a49b5c541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96ff431-l79th" Nov 01 06:28:20.139301 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:20.139267 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96ff431-l79th" Nov 01 06:28:20.517666 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:28:20.517638 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeeb8104_b759_478c_8529_f83a49b5c541.slice/crio-73f78b95c2b71609100d470cbce0fbb638d556b559cc79a85ef7f6a479d2df2a WatchSource:0}: Error finding container 73f78b95c2b71609100d470cbce0fbb638d556b559cc79a85ef7f6a479d2df2a: Status 404 returned error can't find the container with id 73f78b95c2b71609100d470cbce0fbb638d556b559cc79a85ef7f6a479d2df2a Nov 01 06:28:20.517666 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:20.517654 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c96ff431-l79th"] Nov 01 06:28:20.660898 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:20.660859 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96ff431-l79th" event={"ID":"beeb8104-b759-478c-8529-f83a49b5c541","Type":"ContainerStarted","Data":"73f78b95c2b71609100d470cbce0fbb638d556b559cc79a85ef7f6a479d2df2a"} Nov 01 06:28:20.910391 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:20.910346 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:21.670868 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:21.670750 4385 generic.go:356] "Generic (PLEG): container finished" podID="beeb8104-b759-478c-8529-f83a49b5c541" containerID="ce332e32bd102f1c79c5687a1645dcf11cb2488eea9897d44024b3c62a13baf7" exitCode=0 Nov 01 06:28:21.670868 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:21.670815 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96ff431-l79th" event={"ID":"beeb8104-b759-478c-8529-f83a49b5c541","Type":"ContainerDied","Data":"ce332e32bd102f1c79c5687a1645dcf11cb2488eea9897d44024b3c62a13baf7"} Nov 01 06:28:21.740210 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:21.740171 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:21.892499 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:21.892465 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:22.840234 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:22.840209 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96ff431-l79th" Nov 01 06:28:22.952461 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:22.952384 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beeb8104-b759-478c-8529-f83a49b5c541-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beeb8104-b759-478c-8529-f83a49b5c541-maas-api-xks-ca\") pod \"beeb8104-b759-478c-8529-f83a49b5c541\" (UID: \"beeb8104-b759-478c-8529-f83a49b5c541\") " Nov 01 06:28:22.952461 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:22.952451 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beeb8104-b759-478c-8529-f83a49b5c541-kube-api-access-bf5fb\" (UniqueName: \"kubernetes.io/projected/beeb8104-b759-478c-8529-f83a49b5c541-kube-api-access-bf5fb\") pod \"beeb8104-b759-478c-8529-f83a49b5c541\" (UID: \"beeb8104-b759-478c-8529-f83a49b5c541\") " Nov 01 06:28:22.952687 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:22.952499 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beeb8104-b759-478c-8529-f83a49b5c541-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beeb8104-b759-478c-8529-f83a49b5c541-maas-api-openshift-ca\") pod \"beeb8104-b759-478c-8529-f83a49b5c541\" (UID: \"beeb8104-b759-478c-8529-f83a49b5c541\") " Nov 01 06:28:22.952904 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:22.952865 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beeb8104-b759-478c-8529-f83a49b5c541-maas-api-openshift-ca" pod "beeb8104-b759-478c-8529-f83a49b5c541" (UID: "beeb8104-b759-478c-8529-f83a49b5c541"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:28:22.955003 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:22.954975 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beeb8104-b759-478c-8529-f83a49b5c541-maas-api-xks-ca" pod "beeb8104-b759-478c-8529-f83a49b5c541" (UID: "beeb8104-b759-478c-8529-f83a49b5c541"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:28:22.955204 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:22.955179 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beeb8104-b759-478c-8529-f83a49b5c541-kube-api-access-bf5fb" pod "beeb8104-b759-478c-8529-f83a49b5c541" (UID: "beeb8104-b759-478c-8529-f83a49b5c541"). InnerVolumeSpecName "kube-api-access-bf5fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:28:23.053936 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:23.053902 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beeb8104-b759-478c-8529-f83a49b5c541-maas-api-xks-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:23.053936 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:23.053929 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf5fb\" (UniqueName: \"kubernetes.io/projected/beeb8104-b759-478c-8529-f83a49b5c541-kube-api-access-bf5fb\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:23.053936 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:23.053940 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beeb8104-b759-478c-8529-f83a49b5c541-maas-api-openshift-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:23.301802 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:23.301712 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:23.698860 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:23.692879 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96ff431-l79th" event={"ID":"beeb8104-b759-478c-8529-f83a49b5c541","Type":"ContainerDied","Data":"73f78b95c2b71609100d470cbce0fbb638d556b559cc79a85ef7f6a479d2df2a"} Nov 01 06:28:23.698860 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:23.692980 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73f78b95c2b71609100d470cbce0fbb638d556b559cc79a85ef7f6a479d2df2a" Nov 01 06:28:23.698860 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:23.693053 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96ff431-l79th" Nov 01 06:28:23.797268 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:23.797231 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:25.297743 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:25.297707 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:25.893700 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:25.893663 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:26.040881 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:26.039754 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:26.260993 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:26.260904 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:27.177216 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:27.177184 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:27.370065 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:27.369815 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:27.409619 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:27.409552 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx" Nov 01 06:28:27.508497 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:27.508456 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7731ee91-483d-4396-8146-68e18afdd3c4-kube-api-access-7jhjv\" (UniqueName: \"kubernetes.io/projected/7731ee91-483d-4396-8146-68e18afdd3c4-kube-api-access-7jhjv\") pod \"7731ee91-483d-4396-8146-68e18afdd3c4\" (UID: \"7731ee91-483d-4396-8146-68e18afdd3c4\") " Nov 01 06:28:27.508497 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:27.508501 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7731ee91-483d-4396-8146-68e18afdd3c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7731ee91-483d-4396-8146-68e18afdd3c4-plugins-config-volume\") pod \"7731ee91-483d-4396-8146-68e18afdd3c4\" (UID: \"7731ee91-483d-4396-8146-68e18afdd3c4\") " Nov 01 06:28:27.508972 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:27.508947 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7731ee91-483d-4396-8146-68e18afdd3c4-plugins-config-volume" pod "7731ee91-483d-4396-8146-68e18afdd3c4" (UID: "7731ee91-483d-4396-8146-68e18afdd3c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:28:27.510849 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:27.510802 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7731ee91-483d-4396-8146-68e18afdd3c4-kube-api-access-7jhjv" pod "7731ee91-483d-4396-8146-68e18afdd3c4" (UID: "7731ee91-483d-4396-8146-68e18afdd3c4"). InnerVolumeSpecName "kube-api-access-7jhjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:28:27.609438 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:27.609406 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jhjv\" (UniqueName: \"kubernetes.io/projected/7731ee91-483d-4396-8146-68e18afdd3c4-kube-api-access-7jhjv\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:27.609438 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:27.609434 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7731ee91-483d-4396-8146-68e18afdd3c4-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:27.726787 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:27.726702 4385 generic.go:356] "Generic (PLEG): container finished" podID="7731ee91-483d-4396-8146-68e18afdd3c4" containerID="53686ef8a8bd30cc543fd88da987803e64ea51365fff209b4de681a7e29d4139" exitCode=0 Nov 01 06:28:27.726787 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:27.726751 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx" Nov 01 06:28:27.727127 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:27.726783 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx" event={"ID":"7731ee91-483d-4396-8146-68e18afdd3c4","Type":"ContainerDied","Data":"53686ef8a8bd30cc543fd88da987803e64ea51365fff209b4de681a7e29d4139"} Nov 01 06:28:27.727127 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:27.726860 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx" event={"ID":"7731ee91-483d-4396-8146-68e18afdd3c4","Type":"ContainerDied","Data":"fc7710402bc16a0661c97dbd9c051061060c4e9d8ded4d94365b45a9626a513a"} Nov 01 06:28:27.727127 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:27.726887 4385 scope.go:122] "RemoveContainer" containerID="53686ef8a8bd30cc543fd88da987803e64ea51365fff209b4de681a7e29d4139" Nov 01 06:28:27.737180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:27.737159 4385 scope.go:122] "RemoveContainer" containerID="53686ef8a8bd30cc543fd88da987803e64ea51365fff209b4de681a7e29d4139" Nov 01 06:28:27.737483 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:28:27.737462 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53686ef8a8bd30cc543fd88da987803e64ea51365fff209b4de681a7e29d4139\": container with ID starting with 53686ef8a8bd30cc543fd88da987803e64ea51365fff209b4de681a7e29d4139 not found: ID does not exist" containerID="53686ef8a8bd30cc543fd88da987803e64ea51365fff209b4de681a7e29d4139" Nov 01 06:28:27.737533 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:27.737492 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53686ef8a8bd30cc543fd88da987803e64ea51365fff209b4de681a7e29d4139"} err="failed to get container status \"53686ef8a8bd30cc543fd88da987803e64ea51365fff209b4de681a7e29d4139\": rpc error: code = NotFound desc = could not find container \"53686ef8a8bd30cc543fd88da987803e64ea51365fff209b4de681a7e29d4139\": container with ID starting with 53686ef8a8bd30cc543fd88da987803e64ea51365fff209b4de681a7e29d4139 not found: ID does not exist" Nov 01 06:28:27.892661 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:27.892615 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx"] Nov 01 06:28:27.896561 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:27.896530 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ef9b55-785fcbf955-hwknx"] Nov 01 06:28:28.384753 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:28.384721 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7731ee91-483d-4396-8146-68e18afdd3c4" path="/var/lib/kubelet/pods/7731ee91-483d-4396-8146-68e18afdd3c4/volumes" Nov 01 06:28:29.637102 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:29.637061 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:29.843318 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:29.843277 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:31.201553 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:31.201515 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:32.972609 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:32.972574 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:33.155791 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:33.155755 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:33.369407 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:33.369368 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48"] Nov 01 06:28:33.369867 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:33.369827 4385 cpu_manager.go:414] "containerMap: removing container" podUID="beeb8104-b759-478c-8529-f83a49b5c541" containerName="revoke-keys" Nov 01 06:28:33.369867 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:33.369860 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beeb8104-b759-478c-8529-f83a49b5c541" containerName="revoke-keys" Nov 01 06:28:33.370029 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:33.369877 4385 cpu_manager.go:414] "containerMap: removing container" podUID="7731ee91-483d-4396-8146-68e18afdd3c4" containerName="payload-processing" Nov 01 06:28:33.370029 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:33.369883 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7731ee91-483d-4396-8146-68e18afdd3c4" containerName="payload-processing" Nov 01 06:28:33.370029 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:33.369971 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="beeb8104-b759-478c-8529-f83a49b5c541" containerName="revoke-keys" Nov 01 06:28:33.370029 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:33.369980 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="7731ee91-483d-4396-8146-68e18afdd3c4" containerName="payload-processing" Nov 01 06:28:33.374750 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:33.374731 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48" Nov 01 06:28:33.433772 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:33.433745 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48"] Nov 01 06:28:33.478553 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:33.478521 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-e7b2a3\"" Nov 01 06:28:33.478775 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:33.478532 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-e7b2a3-dockercfg-wgw28\"" Nov 01 06:28:33.539912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:33.539880 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca2f250-9e02-4618-a254-da9e55299134-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48\" (UID: \"fca2f250-9e02-4618-a254-da9e55299134\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48" Nov 01 06:28:33.543559 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:33.543527 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7kgb\" (UniqueName: \"kubernetes.io/projected/fca2f250-9e02-4618-a254-da9e55299134-kube-api-access-c7kgb\") pod \"payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48\" (UID: \"fca2f250-9e02-4618-a254-da9e55299134\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48" Nov 01 06:28:33.645196 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:33.645126 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7kgb\" (UniqueName: \"kubernetes.io/projected/fca2f250-9e02-4618-a254-da9e55299134-kube-api-access-c7kgb\") pod \"payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48\" (UID: \"fca2f250-9e02-4618-a254-da9e55299134\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48" Nov 01 06:28:33.645345 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:33.645254 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca2f250-9e02-4618-a254-da9e55299134-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48\" (UID: \"fca2f250-9e02-4618-a254-da9e55299134\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48" Nov 01 06:28:33.645953 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:33.645932 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca2f250-9e02-4618-a254-da9e55299134-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48\" (UID: \"fca2f250-9e02-4618-a254-da9e55299134\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48" Nov 01 06:28:33.701770 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:33.701741 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7kgb\" (UniqueName: \"kubernetes.io/projected/fca2f250-9e02-4618-a254-da9e55299134-kube-api-access-c7kgb\") pod \"payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48\" (UID: \"fca2f250-9e02-4618-a254-da9e55299134\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48" Nov 01 06:28:33.988292 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:33.988191 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48" Nov 01 06:28:34.369736 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:34.369709 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48"] Nov 01 06:28:34.369988 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:28:34.369964 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca2f250_9e02_4618_a254_da9e55299134.slice/crio-2d688b4e17d191ed2c24d3eaaf7000ae8845148735cc0fdec459945e50852f83 WatchSource:0}: Error finding container 2d688b4e17d191ed2c24d3eaaf7000ae8845148735cc0fdec459945e50852f83: Status 404 returned error can't find the container with id 2d688b4e17d191ed2c24d3eaaf7000ae8845148735cc0fdec459945e50852f83 Nov 01 06:28:34.785289 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:34.785182 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48" event={"ID":"fca2f250-9e02-4618-a254-da9e55299134","Type":"ContainerStarted","Data":"7c526a2cef825c0dab8745736e7b2ec62744e71fc6c88cc7042eef07788d4f17"} Nov 01 06:28:34.785289 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:34.785240 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48" event={"ID":"fca2f250-9e02-4618-a254-da9e55299134","Type":"ContainerStarted","Data":"2d688b4e17d191ed2c24d3eaaf7000ae8845148735cc0fdec459945e50852f83"} Nov 01 06:28:34.785570 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:34.785398 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48" Nov 01 06:28:34.906587 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:34.906522 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48" podStartSLOduration=1.9064998549999999 podStartE2EDuration="1.906499855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:28:34.897588105 +0000 UTC m=+1124.661387962" watchObservedRunningTime="2026-11-01 06:28:34.906499855 +0000 UTC m=+1124.670299714" Nov 01 06:28:36.243296 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:36.243256 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:36.779715 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:28:36.779677 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:28:37.242555 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:37.242525 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6309d2cc-9p2c7"] Nov 01 06:28:37.246323 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:37.246306 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6309d2cc-9p2c7" Nov 01 06:28:37.302489 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:37.302464 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6309d2cc-9p2c7"] Nov 01 06:28:37.346563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:37.346532 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:28:37.346719 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:37.346576 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\"" Nov 01 06:28:37.386797 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:37.386768 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/139e0fdf-2af1-473f-8203-7f6ec40c2814-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6309d2cc-9p2c7\" (UID: \"139e0fdf-2af1-473f-8203-7f6ec40c2814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6309d2cc-9p2c7" Nov 01 06:28:37.386974 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:37.386810 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr5w9\" (UniqueName: \"kubernetes.io/projected/139e0fdf-2af1-473f-8203-7f6ec40c2814-kube-api-access-mr5w9\") pod \"maas-api-revoke-sub-6309d2cc-9p2c7\" (UID: \"139e0fdf-2af1-473f-8203-7f6ec40c2814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6309d2cc-9p2c7" Nov 01 06:28:37.387051 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:37.387032 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/139e0fdf-2af1-473f-8203-7f6ec40c2814-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6309d2cc-9p2c7\" (UID: \"139e0fdf-2af1-473f-8203-7f6ec40c2814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6309d2cc-9p2c7" Nov 01 06:28:37.488458 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:37.488419 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/139e0fdf-2af1-473f-8203-7f6ec40c2814-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6309d2cc-9p2c7\" (UID: \"139e0fdf-2af1-473f-8203-7f6ec40c2814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6309d2cc-9p2c7" Nov 01 06:28:37.488609 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:37.488497 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/139e0fdf-2af1-473f-8203-7f6ec40c2814-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6309d2cc-9p2c7\" (UID: \"139e0fdf-2af1-473f-8203-7f6ec40c2814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6309d2cc-9p2c7" Nov 01 06:28:37.488609 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:37.488541 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr5w9\" (UniqueName: \"kubernetes.io/projected/139e0fdf-2af1-473f-8203-7f6ec40c2814-kube-api-access-mr5w9\") pod \"maas-api-revoke-sub-6309d2cc-9p2c7\" (UID: \"139e0fdf-2af1-473f-8203-7f6ec40c2814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6309d2cc-9p2c7" Nov 01 06:28:37.489206 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:37.489187 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/139e0fdf-2af1-473f-8203-7f6ec40c2814-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6309d2cc-9p2c7\" (UID: \"139e0fdf-2af1-473f-8203-7f6ec40c2814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6309d2cc-9p2c7" Nov 01 06:28:37.491067 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:37.491041 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/139e0fdf-2af1-473f-8203-7f6ec40c2814-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6309d2cc-9p2c7\" (UID: \"139e0fdf-2af1-473f-8203-7f6ec40c2814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6309d2cc-9p2c7" Nov 01 06:28:37.545373 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:37.545300 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr5w9\" (UniqueName: \"kubernetes.io/projected/139e0fdf-2af1-473f-8203-7f6ec40c2814-kube-api-access-mr5w9\") pod \"maas-api-revoke-sub-6309d2cc-9p2c7\" (UID: \"139e0fdf-2af1-473f-8203-7f6ec40c2814\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6309d2cc-9p2c7" Nov 01 06:28:37.560415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:37.560388 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6309d2cc-9p2c7" Nov 01 06:28:37.937762 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:28:37.937731 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139e0fdf_2af1_473f_8203_7f6ec40c2814.slice/crio-9134fe32aab36dea8e692fb65bb155159b3afffa4dfa9ce1580f8db1408e0f16 WatchSource:0}: Error finding container 9134fe32aab36dea8e692fb65bb155159b3afffa4dfa9ce1580f8db1408e0f16: Status 404 returned error can't find the container with id 9134fe32aab36dea8e692fb65bb155159b3afffa4dfa9ce1580f8db1408e0f16 Nov 01 06:28:37.938197 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:37.938176 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6309d2cc-9p2c7"] Nov 01 06:28:38.812661 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:38.812630 4385 generic.go:356] "Generic (PLEG): container finished" podID="139e0fdf-2af1-473f-8203-7f6ec40c2814" containerID="b3a8fb4b942112e862fa71834d73a60bfed5d94da7423304e8b534e46bead324" exitCode=0 Nov 01 06:28:38.813032 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:38.812709 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6309d2cc-9p2c7" event={"ID":"139e0fdf-2af1-473f-8203-7f6ec40c2814","Type":"ContainerDied","Data":"b3a8fb4b942112e862fa71834d73a60bfed5d94da7423304e8b534e46bead324"} Nov 01 06:28:38.813032 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:38.812761 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6309d2cc-9p2c7" event={"ID":"139e0fdf-2af1-473f-8203-7f6ec40c2814","Type":"ContainerStarted","Data":"9134fe32aab36dea8e692fb65bb155159b3afffa4dfa9ce1580f8db1408e0f16"} Nov 01 06:28:39.987019 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:39.986997 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6309d2cc-9p2c7" Nov 01 06:28:40.016265 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:40.016235 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/139e0fdf-2af1-473f-8203-7f6ec40c2814-kube-api-access-mr5w9\" (UniqueName: \"kubernetes.io/projected/139e0fdf-2af1-473f-8203-7f6ec40c2814-kube-api-access-mr5w9\") pod \"139e0fdf-2af1-473f-8203-7f6ec40c2814\" (UID: \"139e0fdf-2af1-473f-8203-7f6ec40c2814\") " Nov 01 06:28:40.016419 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:40.016277 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/139e0fdf-2af1-473f-8203-7f6ec40c2814-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/139e0fdf-2af1-473f-8203-7f6ec40c2814-maas-api-xks-ca\") pod \"139e0fdf-2af1-473f-8203-7f6ec40c2814\" (UID: \"139e0fdf-2af1-473f-8203-7f6ec40c2814\") " Nov 01 06:28:40.016419 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:40.016332 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/139e0fdf-2af1-473f-8203-7f6ec40c2814-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/139e0fdf-2af1-473f-8203-7f6ec40c2814-maas-api-openshift-ca\") pod \"139e0fdf-2af1-473f-8203-7f6ec40c2814\" (UID: \"139e0fdf-2af1-473f-8203-7f6ec40c2814\") " Nov 01 06:28:40.016778 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:40.016748 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/139e0fdf-2af1-473f-8203-7f6ec40c2814-maas-api-openshift-ca" pod "139e0fdf-2af1-473f-8203-7f6ec40c2814" (UID: "139e0fdf-2af1-473f-8203-7f6ec40c2814"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:28:40.018522 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:40.018492 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/139e0fdf-2af1-473f-8203-7f6ec40c2814-maas-api-xks-ca" pod "139e0fdf-2af1-473f-8203-7f6ec40c2814" (UID: "139e0fdf-2af1-473f-8203-7f6ec40c2814"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:28:40.018604 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:40.018567 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/139e0fdf-2af1-473f-8203-7f6ec40c2814-kube-api-access-mr5w9" pod "139e0fdf-2af1-473f-8203-7f6ec40c2814" (UID: "139e0fdf-2af1-473f-8203-7f6ec40c2814"). InnerVolumeSpecName "kube-api-access-mr5w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:28:40.118544 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:40.118499 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr5w9\" (UniqueName: \"kubernetes.io/projected/139e0fdf-2af1-473f-8203-7f6ec40c2814-kube-api-access-mr5w9\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:40.118544 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:40.118547 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/139e0fdf-2af1-473f-8203-7f6ec40c2814-maas-api-xks-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:40.118813 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:40.118564 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/139e0fdf-2af1-473f-8203-7f6ec40c2814-maas-api-openshift-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:40.248164 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:40.248119 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:40.826909 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:40.826878 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6309d2cc-9p2c7" Nov 01 06:28:40.826909 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:40.826896 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6309d2cc-9p2c7" event={"ID":"139e0fdf-2af1-473f-8203-7f6ec40c2814","Type":"ContainerDied","Data":"9134fe32aab36dea8e692fb65bb155159b3afffa4dfa9ce1580f8db1408e0f16"} Nov 01 06:28:40.827110 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:40.826941 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9134fe32aab36dea8e692fb65bb155159b3afffa4dfa9ce1580f8db1408e0f16" Nov 01 06:28:41.092261 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:41.092184 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495f7364-fdhmn"] Nov 01 06:28:41.092660 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:41.092647 4385 cpu_manager.go:414] "containerMap: removing container" podUID="139e0fdf-2af1-473f-8203-7f6ec40c2814" containerName="revoke-keys" Nov 01 06:28:41.092706 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:41.092662 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="139e0fdf-2af1-473f-8203-7f6ec40c2814" containerName="revoke-keys" Nov 01 06:28:41.092786 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:41.092776 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="139e0fdf-2af1-473f-8203-7f6ec40c2814" containerName="revoke-keys" Nov 01 06:28:41.096095 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:41.096079 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495f7364-fdhmn" Nov 01 06:28:41.129608 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:41.129581 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf5sc\" (UniqueName: \"kubernetes.io/projected/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5-kube-api-access-mf5sc\") pod \"maas-api-revoke-sub-495f7364-fdhmn\" (UID: \"e7cf5df4-dd07-4bb3-a7dd-08215f5643f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495f7364-fdhmn" Nov 01 06:28:41.129790 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:41.129758 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-495f7364-fdhmn\" (UID: \"e7cf5df4-dd07-4bb3-a7dd-08215f5643f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495f7364-fdhmn" Nov 01 06:28:41.129965 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:41.129948 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-495f7364-fdhmn\" (UID: \"e7cf5df4-dd07-4bb3-a7dd-08215f5643f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495f7364-fdhmn" Nov 01 06:28:41.151795 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:41.151769 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495f7364-fdhmn"] Nov 01 06:28:41.165486 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:41.165457 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:28:41.165624 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:41.165545 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\"" Nov 01 06:28:41.231366 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:41.231338 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-495f7364-fdhmn\" (UID: \"e7cf5df4-dd07-4bb3-a7dd-08215f5643f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495f7364-fdhmn" Nov 01 06:28:41.231548 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:41.231401 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf5sc\" (UniqueName: \"kubernetes.io/projected/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5-kube-api-access-mf5sc\") pod \"maas-api-revoke-sub-495f7364-fdhmn\" (UID: \"e7cf5df4-dd07-4bb3-a7dd-08215f5643f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495f7364-fdhmn" Nov 01 06:28:41.231548 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:41.231434 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-495f7364-fdhmn\" (UID: \"e7cf5df4-dd07-4bb3-a7dd-08215f5643f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495f7364-fdhmn" Nov 01 06:28:41.232089 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:41.232052 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-495f7364-fdhmn\" (UID: \"e7cf5df4-dd07-4bb3-a7dd-08215f5643f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495f7364-fdhmn" Nov 01 06:28:41.233807 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:41.233780 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-495f7364-fdhmn\" (UID: \"e7cf5df4-dd07-4bb3-a7dd-08215f5643f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495f7364-fdhmn" Nov 01 06:28:41.287878 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:41.287823 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf5sc\" (UniqueName: \"kubernetes.io/projected/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5-kube-api-access-mf5sc\") pod \"maas-api-revoke-sub-495f7364-fdhmn\" (UID: \"e7cf5df4-dd07-4bb3-a7dd-08215f5643f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495f7364-fdhmn" Nov 01 06:28:41.411569 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:41.411486 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495f7364-fdhmn" Nov 01 06:28:41.801135 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:41.800995 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495f7364-fdhmn"] Nov 01 06:28:41.842617 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:41.842589 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495f7364-fdhmn" event={"ID":"e7cf5df4-dd07-4bb3-a7dd-08215f5643f5","Type":"ContainerStarted","Data":"3cdcd51ef792428ca9866bf4668dfe33b6c03ba67fc621a77b1b7dfa2e84844b"} Nov 01 06:28:42.527225 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:42.527107 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:42.851562 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:42.851526 4385 generic.go:356] "Generic (PLEG): container finished" podID="e7cf5df4-dd07-4bb3-a7dd-08215f5643f5" containerID="35cc4a7e764a28bedb8271d3c8991b0f161bb9a7dfe717b62354c63c6c8b57fd" exitCode=0 Nov 01 06:28:42.851738 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:42.851619 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495f7364-fdhmn" event={"ID":"e7cf5df4-dd07-4bb3-a7dd-08215f5643f5","Type":"ContainerDied","Data":"35cc4a7e764a28bedb8271d3c8991b0f161bb9a7dfe717b62354c63c6c8b57fd"} Nov 01 06:28:42.968962 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:42.968931 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5"] Nov 01 06:28:42.973097 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:42.973075 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.052732 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.052650 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51691383-996f-4d97-b9ab-e88457d392fc-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.052732 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.052697 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxtds\" (UniqueName: \"kubernetes.io/projected/51691383-996f-4d97-b9ab-e88457d392fc-kube-api-access-nxtds\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.052732 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.052718 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.052968 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.052868 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51691383-996f-4d97-b9ab-e88457d392fc-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.052968 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.052912 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.053088 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.053060 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.053127 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.053115 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.053245 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.053206 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.053373 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.053266 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51691383-996f-4d97-b9ab-e88457d392fc-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.072603 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.072569 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5"] Nov 01 06:28:43.154970 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.154928 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.155185 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.155004 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51691383-996f-4d97-b9ab-e88457d392fc-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.155185 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.155062 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51691383-996f-4d97-b9ab-e88457d392fc-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.155185 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.155111 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxtds\" (UniqueName: \"kubernetes.io/projected/51691383-996f-4d97-b9ab-e88457d392fc-kube-api-access-nxtds\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.155185 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.155144 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.155557 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.155533 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51691383-996f-4d97-b9ab-e88457d392fc-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.155856 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.155780 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.155856 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.155872 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.156021 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.155952 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.156021 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.155959 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51691383-996f-4d97-b9ab-e88457d392fc-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.156021 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.155988 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.156262 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.156228 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.156393 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.156326 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.156502 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.156481 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.158099 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.158076 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.158462 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.158442 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51691383-996f-4d97-b9ab-e88457d392fc-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.211578 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.211544 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51691383-996f-4d97-b9ab-e88457d392fc-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.211735 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.211604 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxtds\" (UniqueName: \"kubernetes.io/projected/51691383-996f-4d97-b9ab-e88457d392fc-kube-api-access-nxtds\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.286268 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.286238 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:43.714860 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:28:43.714808 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51691383_996f_4d97_b9ab_e88457d392fc.slice/crio-13d01a421bd99be5224463354366529d5b096df59faf53d66e322efb03355640 WatchSource:0}: Error finding container 13d01a421bd99be5224463354366529d5b096df59faf53d66e322efb03355640: Status 404 returned error can't find the container with id 13d01a421bd99be5224463354366529d5b096df59faf53d66e322efb03355640 Nov 01 06:28:43.715409 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.715387 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5"] Nov 01 06:28:43.717623 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.717582 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:28:43.717697 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.717670 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:28:43.717741 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.717715 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:28:43.862125 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.862086 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" event={"ID":"51691383-996f-4d97-b9ab-e88457d392fc","Type":"ContainerStarted","Data":"917842658bf121eb1530e653bef46b2d0a7ae80dfc057068cd27f5e7bde80501"} Nov 01 06:28:43.862303 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.862158 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" event={"ID":"51691383-996f-4d97-b9ab-e88457d392fc","Type":"ContainerStarted","Data":"13d01a421bd99be5224463354366529d5b096df59faf53d66e322efb03355640"} Nov 01 06:28:43.987323 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:43.985379 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" podStartSLOduration=1.985357756 podStartE2EDuration="1.985357756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:28:43.977415173 +0000 UTC m=+1133.741215043" watchObservedRunningTime="2026-11-01 06:28:43.985357756 +0000 UTC m=+1133.749157616" Nov 01 06:28:44.032475 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:44.032451 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495f7364-fdhmn" Nov 01 06:28:44.065692 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:44.065666 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5-kube-api-access-mf5sc\" (UniqueName: \"kubernetes.io/projected/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5-kube-api-access-mf5sc\") pod \"e7cf5df4-dd07-4bb3-a7dd-08215f5643f5\" (UID: \"e7cf5df4-dd07-4bb3-a7dd-08215f5643f5\") " Nov 01 06:28:44.065867 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:44.065718 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5-maas-api-openshift-ca\") pod \"e7cf5df4-dd07-4bb3-a7dd-08215f5643f5\" (UID: \"e7cf5df4-dd07-4bb3-a7dd-08215f5643f5\") " Nov 01 06:28:44.065867 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:44.065816 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5-maas-api-xks-ca\") pod \"e7cf5df4-dd07-4bb3-a7dd-08215f5643f5\" (UID: \"e7cf5df4-dd07-4bb3-a7dd-08215f5643f5\") " Nov 01 06:28:44.066186 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:44.066161 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5-maas-api-openshift-ca" pod "e7cf5df4-dd07-4bb3-a7dd-08215f5643f5" (UID: "e7cf5df4-dd07-4bb3-a7dd-08215f5643f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:28:44.066348 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:44.066333 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5-maas-api-openshift-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:44.068161 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:44.068136 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5-kube-api-access-mf5sc" pod "e7cf5df4-dd07-4bb3-a7dd-08215f5643f5" (UID: "e7cf5df4-dd07-4bb3-a7dd-08215f5643f5"). InnerVolumeSpecName "kube-api-access-mf5sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:28:44.068161 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:44.068140 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5-maas-api-xks-ca" pod "e7cf5df4-dd07-4bb3-a7dd-08215f5643f5" (UID: "e7cf5df4-dd07-4bb3-a7dd-08215f5643f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:28:44.167466 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:44.167430 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5-maas-api-xks-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:44.167622 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:44.167471 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf5sc\" (UniqueName: \"kubernetes.io/projected/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5-kube-api-access-mf5sc\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:44.287200 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:44.287162 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:44.288731 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:44.288701 4385 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 06:28:44.288909 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:44.288750 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" podUID="51691383-996f-4d97-b9ab-e88457d392fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 06:28:44.879859 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:44.879564 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495f7364-fdhmn" Nov 01 06:28:44.882868 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:44.882712 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495f7364-fdhmn" event={"ID":"e7cf5df4-dd07-4bb3-a7dd-08215f5643f5","Type":"ContainerDied","Data":"3cdcd51ef792428ca9866bf4668dfe33b6c03ba67fc621a77b1b7dfa2e84844b"} Nov 01 06:28:44.882868 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:44.882769 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cdcd51ef792428ca9866bf4668dfe33b6c03ba67fc621a77b1b7dfa2e84844b" Nov 01 06:28:45.288364 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:45.288266 4385 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 06:28:45.288663 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:45.288629 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" podUID="51691383-996f-4d97-b9ab-e88457d392fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 06:28:45.806207 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:45.806179 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48" Nov 01 06:28:46.290857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:46.290187 4385 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 06:28:46.290857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:46.290246 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" podUID="51691383-996f-4d97-b9ab-e88457d392fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 06:28:47.223416 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:47.223366 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:47.292867 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:47.292230 4385 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 06:28:47.292867 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:47.292296 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" podUID="51691383-996f-4d97-b9ab-e88457d392fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 06:28:48.287858 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:48.287550 4385 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 06:28:48.287858 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:48.287612 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" podUID="51691383-996f-4d97-b9ab-e88457d392fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 06:28:49.309090 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:49.301465 4385 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 06:28:49.309090 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:49.301524 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" podUID="51691383-996f-4d97-b9ab-e88457d392fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 06:28:49.871860 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:49.870969 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" podUID="f5f0b488-c1f8-4a7e-9779-ebf4d94ad037" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 01 06:28:50.291281 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:50.291191 4385 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 06:28:50.291550 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:50.291518 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" podUID="51691383-996f-4d97-b9ab-e88457d392fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 06:28:50.417858 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:50.417138 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" podUID="04ae8d7a-c300-4ddc-89a2-7d3400685bda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:28:51.286749 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:51.286714 4385 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 06:28:51.286949 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:51.286776 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" podUID="51691383-996f-4d97-b9ab-e88457d392fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 06:28:52.423575 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:52.423542 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:28:53.197167 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:53.197133 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:53.197555 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:53.197531 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:53.199908 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:53.199887 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:28:57.149892 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:28:57.149852 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:29:01.727598 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:01.727556 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:29:01.865808 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:01.865775 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:29:02.005574 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:02.005489 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:29:02.319943 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:02.319858 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:29:06.782411 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:29:06.782374 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:29:07.045422 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:07.045348 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:29:11.670763 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:11.670727 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:29:11.830440 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:11.830406 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:29:11.975163 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:11.975049 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:29:16.737097 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:16.737061 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:29:17.241268 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:17.241235 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd"] Nov 01 06:29:17.241772 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:17.241754 4385 cpu_manager.go:414] "containerMap: removing container" podUID="e7cf5df4-dd07-4bb3-a7dd-08215f5643f5" containerName="revoke-keys" Nov 01 06:29:17.241888 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:17.241774 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7cf5df4-dd07-4bb3-a7dd-08215f5643f5" containerName="revoke-keys" Nov 01 06:29:17.241989 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:17.241976 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7cf5df4-dd07-4bb3-a7dd-08215f5643f5" containerName="revoke-keys" Nov 01 06:29:17.245589 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:17.245569 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd" Nov 01 06:29:17.301361 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:17.301330 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd"] Nov 01 06:29:17.345845 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:17.345806 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:29:17.345845 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:17.345820 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\"" Nov 01 06:29:17.367163 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:17.367132 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2d34f08-dd95-446b-ac13-337294f74d94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9795dbcb-58lbd\" (UID: \"f2d34f08-dd95-446b-ac13-337294f74d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd" Nov 01 06:29:17.367326 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:17.367180 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttrxd\" (UniqueName: \"kubernetes.io/projected/f2d34f08-dd95-446b-ac13-337294f74d94-kube-api-access-ttrxd\") pod \"maas-api-revoke-sub-9795dbcb-58lbd\" (UID: \"f2d34f08-dd95-446b-ac13-337294f74d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd" Nov 01 06:29:17.367371 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:17.367357 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2d34f08-dd95-446b-ac13-337294f74d94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9795dbcb-58lbd\" (UID: \"f2d34f08-dd95-446b-ac13-337294f74d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd" Nov 01 06:29:17.467985 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:17.467948 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2d34f08-dd95-446b-ac13-337294f74d94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9795dbcb-58lbd\" (UID: \"f2d34f08-dd95-446b-ac13-337294f74d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd" Nov 01 06:29:17.468166 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:17.468004 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttrxd\" (UniqueName: \"kubernetes.io/projected/f2d34f08-dd95-446b-ac13-337294f74d94-kube-api-access-ttrxd\") pod \"maas-api-revoke-sub-9795dbcb-58lbd\" (UID: \"f2d34f08-dd95-446b-ac13-337294f74d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd" Nov 01 06:29:17.468166 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:17.468128 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2d34f08-dd95-446b-ac13-337294f74d94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9795dbcb-58lbd\" (UID: \"f2d34f08-dd95-446b-ac13-337294f74d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd" Nov 01 06:29:17.468746 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:17.468728 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2d34f08-dd95-446b-ac13-337294f74d94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9795dbcb-58lbd\" (UID: \"f2d34f08-dd95-446b-ac13-337294f74d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd" Nov 01 06:29:17.470487 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:17.470463 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2d34f08-dd95-446b-ac13-337294f74d94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9795dbcb-58lbd\" (UID: \"f2d34f08-dd95-446b-ac13-337294f74d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd" Nov 01 06:29:17.524362 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:17.524284 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttrxd\" (UniqueName: \"kubernetes.io/projected/f2d34f08-dd95-446b-ac13-337294f74d94-kube-api-access-ttrxd\") pod \"maas-api-revoke-sub-9795dbcb-58lbd\" (UID: \"f2d34f08-dd95-446b-ac13-337294f74d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd" Nov 01 06:29:17.558782 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:17.558748 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd" Nov 01 06:29:17.939467 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:29:17.939433 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d34f08_dd95_446b_ac13_337294f74d94.slice/crio-a39df352a544bb511f73a4dfca611838a5df29bc328a6dbd6f06037e40ce125b WatchSource:0}: Error finding container a39df352a544bb511f73a4dfca611838a5df29bc328a6dbd6f06037e40ce125b: Status 404 returned error can't find the container with id a39df352a544bb511f73a4dfca611838a5df29bc328a6dbd6f06037e40ce125b Nov 01 06:29:17.939944 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:17.939543 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd"] Nov 01 06:29:18.372792 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:18.372753 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd" event={"ID":"f2d34f08-dd95-446b-ac13-337294f74d94","Type":"ContainerStarted","Data":"2e12480436b293cef633883dab2424f4c773827e1854cc496c00869228ca2a4c"} Nov 01 06:29:18.373012 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:18.372823 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd" event={"ID":"f2d34f08-dd95-446b-ac13-337294f74d94","Type":"ContainerStarted","Data":"a39df352a544bb511f73a4dfca611838a5df29bc328a6dbd6f06037e40ce125b"} Nov 01 06:29:18.485905 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:18.485819 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd" podStartSLOduration=1.485805019 podStartE2EDuration="1.485805019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:29:18.484088244 +0000 UTC m=+1168.247888102" watchObservedRunningTime="2026-11-01 06:29:18.485805019 +0000 UTC m=+1168.249604876" Nov 01 06:29:19.382204 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:19.382168 4385 generic.go:356] "Generic (PLEG): container finished" podID="f2d34f08-dd95-446b-ac13-337294f74d94" containerID="2e12480436b293cef633883dab2424f4c773827e1854cc496c00869228ca2a4c" exitCode=0 Nov 01 06:29:19.382599 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:19.382300 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd" event={"ID":"f2d34f08-dd95-446b-ac13-337294f74d94","Type":"ContainerDied","Data":"2e12480436b293cef633883dab2424f4c773827e1854cc496c00869228ca2a4c"} Nov 01 06:29:20.668883 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:20.668790 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd" Nov 01 06:29:20.746859 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:20.745292 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2d34f08-dd95-446b-ac13-337294f74d94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2d34f08-dd95-446b-ac13-337294f74d94-maas-api-xks-ca\") pod \"f2d34f08-dd95-446b-ac13-337294f74d94\" (UID: \"f2d34f08-dd95-446b-ac13-337294f74d94\") " Nov 01 06:29:20.746859 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:20.745407 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2d34f08-dd95-446b-ac13-337294f74d94-kube-api-access-ttrxd\" (UniqueName: \"kubernetes.io/projected/f2d34f08-dd95-446b-ac13-337294f74d94-kube-api-access-ttrxd\") pod \"f2d34f08-dd95-446b-ac13-337294f74d94\" (UID: \"f2d34f08-dd95-446b-ac13-337294f74d94\") " Nov 01 06:29:20.746859 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:20.745478 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2d34f08-dd95-446b-ac13-337294f74d94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2d34f08-dd95-446b-ac13-337294f74d94-maas-api-openshift-ca\") pod \"f2d34f08-dd95-446b-ac13-337294f74d94\" (UID: \"f2d34f08-dd95-446b-ac13-337294f74d94\") " Nov 01 06:29:20.746859 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:20.746315 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d34f08-dd95-446b-ac13-337294f74d94-maas-api-openshift-ca" pod "f2d34f08-dd95-446b-ac13-337294f74d94" (UID: "f2d34f08-dd95-446b-ac13-337294f74d94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:29:20.754714 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:20.754683 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2d34f08-dd95-446b-ac13-337294f74d94-maas-api-xks-ca" pod "f2d34f08-dd95-446b-ac13-337294f74d94" (UID: "f2d34f08-dd95-446b-ac13-337294f74d94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:29:20.756336 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:20.756304 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d34f08-dd95-446b-ac13-337294f74d94-kube-api-access-ttrxd" pod "f2d34f08-dd95-446b-ac13-337294f74d94" (UID: "f2d34f08-dd95-446b-ac13-337294f74d94"). InnerVolumeSpecName "kube-api-access-ttrxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:29:20.871813 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:20.865743 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2d34f08-dd95-446b-ac13-337294f74d94-maas-api-xks-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:20.871813 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:20.865787 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttrxd\" (UniqueName: \"kubernetes.io/projected/f2d34f08-dd95-446b-ac13-337294f74d94-kube-api-access-ttrxd\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:20.871813 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:20.865805 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2d34f08-dd95-446b-ac13-337294f74d94-maas-api-openshift-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:21.400557 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:21.400521 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd" Nov 01 06:29:21.400751 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:21.400592 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd" event={"ID":"f2d34f08-dd95-446b-ac13-337294f74d94","Type":"ContainerDied","Data":"a39df352a544bb511f73a4dfca611838a5df29bc328a6dbd6f06037e40ce125b"} Nov 01 06:29:21.400751 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:21.400643 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a39df352a544bb511f73a4dfca611838a5df29bc328a6dbd6f06037e40ce125b" Nov 01 06:29:21.673933 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:21.673781 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:29:22.047058 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:22.046969 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn"] Nov 01 06:29:22.095937 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:22.095898 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51a6dfc6-fbngn"] Nov 01 06:29:22.386280 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:22.386248 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c96d1a13-d35d-4144-b5cb-090ef28d84f1" path="/var/lib/kubelet/pods/c96d1a13-d35d-4144-b5cb-090ef28d84f1/volumes" Nov 01 06:29:29.964905 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:29.964864 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:29:34.725365 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:34.725328 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:29:36.782035 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:29:36.781987 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:29:39.657673 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:39.657642 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:29:40.620858 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:40.620318 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:29:43.514150 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:43.514106 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:29:44.006106 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:44.006072 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:29:46.617752 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:46.617716 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:29:51.194746 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:51.194706 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:29:51.389754 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:51.389712 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:29:56.044744 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:56.044661 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:29:57.405969 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:57.405917 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" podUID="04ae8d7a-c300-4ddc-89a2-7d3400685bda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 01 06:29:57.478857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:57.478170 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" podUID="d0029faa-cd51-43d8-8c9f-02399257ea72" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 01 06:29:57.545026 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:57.543957 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" podUID="4001c2de-286a-4407-9c74-0cc2492bc8ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:29:57.672709 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:57.668927 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" podUID="27e2ec0f-708a-4ffb-9694-d1030859fc61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:29:59.462396 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:59.462359 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448"] Nov 01 06:29:59.462925 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:59.462897 4385 cpu_manager.go:414] "containerMap: removing container" podUID="f2d34f08-dd95-446b-ac13-337294f74d94" containerName="revoke-keys" Nov 01 06:29:59.462925 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:59.462910 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2d34f08-dd95-446b-ac13-337294f74d94" containerName="revoke-keys" Nov 01 06:29:59.463049 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:59.462986 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2d34f08-dd95-446b-ac13-337294f74d94" containerName="revoke-keys" Nov 01 06:29:59.466515 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:59.466495 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448" Nov 01 06:29:59.513087 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:59.513057 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc828d8-18f2-4c17-935c-87572fa0001f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39d95c9d-xf448\" (UID: \"6dc828d8-18f2-4c17-935c-87572fa0001f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448" Nov 01 06:29:59.513455 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:59.513319 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc828d8-18f2-4c17-935c-87572fa0001f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39d95c9d-xf448\" (UID: \"6dc828d8-18f2-4c17-935c-87572fa0001f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448" Nov 01 06:29:59.513455 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:59.513408 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpb2k\" (UniqueName: \"kubernetes.io/projected/6dc828d8-18f2-4c17-935c-87572fa0001f-kube-api-access-rpb2k\") pod \"maas-api-revoke-sub-39d95c9d-xf448\" (UID: \"6dc828d8-18f2-4c17-935c-87572fa0001f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448" Nov 01 06:29:59.524268 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:59.524242 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448"] Nov 01 06:29:59.567237 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:59.567208 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:29:59.567237 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:59.567219 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\"" Nov 01 06:29:59.614728 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:59.614697 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc828d8-18f2-4c17-935c-87572fa0001f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39d95c9d-xf448\" (UID: \"6dc828d8-18f2-4c17-935c-87572fa0001f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448" Nov 01 06:29:59.614901 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:59.614750 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpb2k\" (UniqueName: \"kubernetes.io/projected/6dc828d8-18f2-4c17-935c-87572fa0001f-kube-api-access-rpb2k\") pod \"maas-api-revoke-sub-39d95c9d-xf448\" (UID: \"6dc828d8-18f2-4c17-935c-87572fa0001f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448" Nov 01 06:29:59.614901 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:59.614883 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc828d8-18f2-4c17-935c-87572fa0001f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39d95c9d-xf448\" (UID: \"6dc828d8-18f2-4c17-935c-87572fa0001f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448" Nov 01 06:29:59.615476 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:59.615456 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc828d8-18f2-4c17-935c-87572fa0001f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39d95c9d-xf448\" (UID: \"6dc828d8-18f2-4c17-935c-87572fa0001f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448" Nov 01 06:29:59.617389 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:59.617369 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc828d8-18f2-4c17-935c-87572fa0001f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39d95c9d-xf448\" (UID: \"6dc828d8-18f2-4c17-935c-87572fa0001f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448" Nov 01 06:29:59.671079 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:59.671049 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpb2k\" (UniqueName: \"kubernetes.io/projected/6dc828d8-18f2-4c17-935c-87572fa0001f-kube-api-access-rpb2k\") pod \"maas-api-revoke-sub-39d95c9d-xf448\" (UID: \"6dc828d8-18f2-4c17-935c-87572fa0001f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448" Nov 01 06:29:59.781482 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:29:59.781395 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448" Nov 01 06:30:00.166485 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:30:00.166456 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc828d8_18f2_4c17_935c_87572fa0001f.slice/crio-d42b9e0aba40767c1dc44a44244bc282482b859390d387113ab8c08dbdc75a93 WatchSource:0}: Error finding container d42b9e0aba40767c1dc44a44244bc282482b859390d387113ab8c08dbdc75a93: Status 404 returned error can't find the container with id d42b9e0aba40767c1dc44a44244bc282482b859390d387113ab8c08dbdc75a93 Nov 01 06:30:00.168197 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.168167 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448"] Nov 01 06:30:00.178751 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.178725 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr"] Nov 01 06:30:00.184016 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.183991 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" Nov 01 06:30:00.184702 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.184668 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v"] Nov 01 06:30:00.189081 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.189057 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v" Nov 01 06:30:00.193305 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.193285 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786"] Nov 01 06:30:00.197164 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.197146 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" Nov 01 06:30:00.201189 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.201167 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc"] Nov 01 06:30:00.206035 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.204855 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" Nov 01 06:30:00.282174 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.282131 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr"] Nov 01 06:30:00.314567 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.314528 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v"] Nov 01 06:30:00.322603 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.322570 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786"] Nov 01 06:30:00.323164 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.323143 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7ltf\" (UniqueName: \"kubernetes.io/projected/32527ca5-6b0f-42c3-a007-da447e507f15-kube-api-access-s7ltf\") pod \"maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786\" (UID: \"32527ca5-6b0f-42c3-a007-da447e507f15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" Nov 01 06:30:00.323298 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.323232 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdblx\" (UniqueName: \"kubernetes.io/projected/6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7-kube-api-access-zdblx\") pod \"maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc\" (UID: \"6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" Nov 01 06:30:00.323298 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.323258 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e6f2ec-efdd-4594-98f0-6ec213f5b29c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891910-xw57v\" (UID: \"03e6f2ec-efdd-4594-98f0-6ec213f5b29c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v" Nov 01 06:30:00.323298 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.323283 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc17776b-6027-4263-b311-5b6c52ce8fc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr\" (UID: \"dc17776b-6027-4263-b311-5b6c52ce8fc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" Nov 01 06:30:00.323475 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.323305 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkhvf\" (UniqueName: \"kubernetes.io/projected/03e6f2ec-efdd-4594-98f0-6ec213f5b29c-kube-api-access-hkhvf\") pod \"maas-api-key-cleanup-29891910-xw57v\" (UID: \"03e6f2ec-efdd-4594-98f0-6ec213f5b29c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v" Nov 01 06:30:00.323475 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.323374 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4clbk\" (UniqueName: \"kubernetes.io/projected/dc17776b-6027-4263-b311-5b6c52ce8fc2-kube-api-access-4clbk\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr\" (UID: \"dc17776b-6027-4263-b311-5b6c52ce8fc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" Nov 01 06:30:00.323475 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.323406 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc\" (UID: \"6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" Nov 01 06:30:00.323475 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.323450 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32527ca5-6b0f-42c3-a007-da447e507f15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786\" (UID: \"32527ca5-6b0f-42c3-a007-da447e507f15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" Nov 01 06:30:00.327408 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.327384 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc"] Nov 01 06:30:00.424674 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.424580 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4clbk\" (UniqueName: \"kubernetes.io/projected/dc17776b-6027-4263-b311-5b6c52ce8fc2-kube-api-access-4clbk\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr\" (UID: \"dc17776b-6027-4263-b311-5b6c52ce8fc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" Nov 01 06:30:00.424898 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.424873 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc\" (UID: \"6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" Nov 01 06:30:00.425014 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.424946 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32527ca5-6b0f-42c3-a007-da447e507f15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786\" (UID: \"32527ca5-6b0f-42c3-a007-da447e507f15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" Nov 01 06:30:00.425014 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.424978 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7ltf\" (UniqueName: \"kubernetes.io/projected/32527ca5-6b0f-42c3-a007-da447e507f15-kube-api-access-s7ltf\") pod \"maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786\" (UID: \"32527ca5-6b0f-42c3-a007-da447e507f15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" Nov 01 06:30:00.425140 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.425123 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdblx\" (UniqueName: \"kubernetes.io/projected/6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7-kube-api-access-zdblx\") pod \"maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc\" (UID: \"6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" Nov 01 06:30:00.425202 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.425154 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e6f2ec-efdd-4594-98f0-6ec213f5b29c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891910-xw57v\" (UID: \"03e6f2ec-efdd-4594-98f0-6ec213f5b29c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v" Nov 01 06:30:00.425202 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.425193 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc17776b-6027-4263-b311-5b6c52ce8fc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr\" (UID: \"dc17776b-6027-4263-b311-5b6c52ce8fc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" Nov 01 06:30:00.425328 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.425230 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkhvf\" (UniqueName: \"kubernetes.io/projected/03e6f2ec-efdd-4594-98f0-6ec213f5b29c-kube-api-access-hkhvf\") pod \"maas-api-key-cleanup-29891910-xw57v\" (UID: \"03e6f2ec-efdd-4594-98f0-6ec213f5b29c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v" Nov 01 06:30:00.425582 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.425559 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc\" (UID: \"6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" Nov 01 06:30:00.425659 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.425610 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32527ca5-6b0f-42c3-a007-da447e507f15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786\" (UID: \"32527ca5-6b0f-42c3-a007-da447e507f15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" Nov 01 06:30:00.425940 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.425913 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e6f2ec-efdd-4594-98f0-6ec213f5b29c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891910-xw57v\" (UID: \"03e6f2ec-efdd-4594-98f0-6ec213f5b29c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v" Nov 01 06:30:00.426046 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.425989 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc17776b-6027-4263-b311-5b6c52ce8fc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr\" (UID: \"dc17776b-6027-4263-b311-5b6c52ce8fc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" Nov 01 06:30:00.481312 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.481278 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7ltf\" (UniqueName: \"kubernetes.io/projected/32527ca5-6b0f-42c3-a007-da447e507f15-kube-api-access-s7ltf\") pod \"maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786\" (UID: \"32527ca5-6b0f-42c3-a007-da447e507f15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" Nov 01 06:30:00.481771 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.481635 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4clbk\" (UniqueName: \"kubernetes.io/projected/dc17776b-6027-4263-b311-5b6c52ce8fc2-kube-api-access-4clbk\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr\" (UID: \"dc17776b-6027-4263-b311-5b6c52ce8fc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" Nov 01 06:30:00.481771 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.481688 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdblx\" (UniqueName: \"kubernetes.io/projected/6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7-kube-api-access-zdblx\") pod \"maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc\" (UID: \"6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" Nov 01 06:30:00.481958 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.481938 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkhvf\" (UniqueName: \"kubernetes.io/projected/03e6f2ec-efdd-4594-98f0-6ec213f5b29c-kube-api-access-hkhvf\") pod \"maas-api-key-cleanup-29891910-xw57v\" (UID: \"03e6f2ec-efdd-4594-98f0-6ec213f5b29c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v" Nov 01 06:30:00.514392 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.514365 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" Nov 01 06:30:00.521932 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.521911 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v" Nov 01 06:30:00.529963 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.529942 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" Nov 01 06:30:00.535157 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.535138 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" Nov 01 06:30:00.854137 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.854097 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448" event={"ID":"6dc828d8-18f2-4c17-935c-87572fa0001f","Type":"ContainerStarted","Data":"767086cabe1a2a85d028531b86bed3182b4ca867e7c223fcd2b80460b491a92b"} Nov 01 06:30:00.854316 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.854153 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448" event={"ID":"6dc828d8-18f2-4c17-935c-87572fa0001f","Type":"ContainerStarted","Data":"d42b9e0aba40767c1dc44a44244bc282482b859390d387113ab8c08dbdc75a93"} Nov 01 06:30:00.971807 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:00.971714 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448" podStartSLOduration=1.971700002 podStartE2EDuration="1.971700002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:00.967367739 +0000 UTC m=+1210.731167597" watchObservedRunningTime="2026-11-01 06:30:00.971700002 +0000 UTC m=+1210.735499859" Nov 01 06:30:01.000883 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:30:01.000796 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc17776b_6027_4263_b311_5b6c52ce8fc2.slice/crio-e4712199bf777716fcc0551a864be833d9587855b3ccc0be49ee20391821e787 WatchSource:0}: Error finding container e4712199bf777716fcc0551a864be833d9587855b3ccc0be49ee20391821e787: Status 404 returned error can't find the container with id e4712199bf777716fcc0551a864be833d9587855b3ccc0be49ee20391821e787 Nov 01 06:30:01.002881 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:01.002851 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr"] Nov 01 06:30:01.015912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:01.015879 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786"] Nov 01 06:30:01.048650 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:30:01.048618 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e6f2ec_efdd_4594_98f0_6ec213f5b29c.slice/crio-b825b4d05ba7d07f96f493ca981409c90f1a5ee84f1a42536a247dc809d7998f WatchSource:0}: Error finding container b825b4d05ba7d07f96f493ca981409c90f1a5ee84f1a42536a247dc809d7998f: Status 404 returned error can't find the container with id b825b4d05ba7d07f96f493ca981409c90f1a5ee84f1a42536a247dc809d7998f Nov 01 06:30:01.051387 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:01.051362 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v"] Nov 01 06:30:01.054029 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:30:01.054000 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2c76eb_5a4d_4db5_a9f8_63e0e50e1dc7.slice/crio-959b90904fff8d1cc9fa10f564122d4614febfccbed17a90f870722e89d4e151 WatchSource:0}: Error finding container 959b90904fff8d1cc9fa10f564122d4614febfccbed17a90f870722e89d4e151: Status 404 returned error can't find the container with id 959b90904fff8d1cc9fa10f564122d4614febfccbed17a90f870722e89d4e151 Nov 01 06:30:01.055497 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:01.055474 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc"] Nov 01 06:30:01.389602 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:01.389505 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:01.865164 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:01.865076 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v" event={"ID":"03e6f2ec-efdd-4594-98f0-6ec213f5b29c","Type":"ContainerStarted","Data":"9455bb9efe86b293afe78151a548b6217c1cb9d6a25312305d43b6a765550c7e"} Nov 01 06:30:01.865164 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:01.865141 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v" event={"ID":"03e6f2ec-efdd-4594-98f0-6ec213f5b29c","Type":"ContainerStarted","Data":"b825b4d05ba7d07f96f493ca981409c90f1a5ee84f1a42536a247dc809d7998f"} Nov 01 06:30:01.867375 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:01.867349 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" event={"ID":"6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7","Type":"ContainerStarted","Data":"2451ca7988bc7609cc9dbd945b71f43cf37d6f6694d46640b14d46a3bbbc07cd"} Nov 01 06:30:01.867523 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:01.867393 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" event={"ID":"6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7","Type":"ContainerStarted","Data":"959b90904fff8d1cc9fa10f564122d4614febfccbed17a90f870722e89d4e151"} Nov 01 06:30:01.869177 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:01.869158 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" event={"ID":"32527ca5-6b0f-42c3-a007-da447e507f15","Type":"ContainerStarted","Data":"4f3073f7d72bb633d57e42a8bdf30e5fa3d98d9e370090ba9ca48cc52a228158"} Nov 01 06:30:01.869288 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:01.869202 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" event={"ID":"32527ca5-6b0f-42c3-a007-da447e507f15","Type":"ContainerStarted","Data":"d8bb3a8ac9d84557d82038ff8fa0faca991814d0bbfa10e677bc1fe4dc731a1c"} Nov 01 06:30:01.871377 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:01.871086 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" event={"ID":"dc17776b-6027-4263-b311-5b6c52ce8fc2","Type":"ContainerStarted","Data":"990ad84e19afa7da69fe3c521c99176d6567a356abe77a132c79963583f12633"} Nov 01 06:30:01.871377 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:01.871131 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" event={"ID":"dc17776b-6027-4263-b311-5b6c52ce8fc2","Type":"ContainerStarted","Data":"e4712199bf777716fcc0551a864be833d9587855b3ccc0be49ee20391821e787"} Nov 01 06:30:01.874057 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:01.874029 4385 generic.go:356] "Generic (PLEG): container finished" podID="6dc828d8-18f2-4c17-935c-87572fa0001f" containerID="767086cabe1a2a85d028531b86bed3182b4ca867e7c223fcd2b80460b491a92b" exitCode=0 Nov 01 06:30:01.874472 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:01.874127 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448" event={"ID":"6dc828d8-18f2-4c17-935c-87572fa0001f","Type":"ContainerDied","Data":"767086cabe1a2a85d028531b86bed3182b4ca867e7c223fcd2b80460b491a92b"} Nov 01 06:30:01.985806 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:01.985743 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v" podStartSLOduration=1.9857225509999998 podStartE2EDuration="1.985722551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:01.980244428 +0000 UTC m=+1211.744044289" watchObservedRunningTime="2026-11-01 06:30:01.985722551 +0000 UTC m=+1211.749522409" Nov 01 06:30:02.096203 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:02.096141 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" podStartSLOduration=2.096121802 podStartE2EDuration="2.096121802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:02.090673741 +0000 UTC m=+1211.854473599" watchObservedRunningTime="2026-11-01 06:30:02.096121802 +0000 UTC m=+1211.859921660" Nov 01 06:30:02.206442 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:02.206388 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" podStartSLOduration=2.206369008 podStartE2EDuration="2.206369008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:02.20106649 +0000 UTC m=+1211.964866348" watchObservedRunningTime="2026-11-01 06:30:02.206369008 +0000 UTC m=+1211.970168866" Nov 01 06:30:02.425078 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:02.424976 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" podStartSLOduration=2.424959458 podStartE2EDuration="2.424959458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:02.424103007 +0000 UTC m=+1212.187902865" watchObservedRunningTime="2026-11-01 06:30:02.424959458 +0000 UTC m=+1212.188759315" Nov 01 06:30:02.887930 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:02.887894 4385 generic.go:356] "Generic (PLEG): container finished" podID="03e6f2ec-efdd-4594-98f0-6ec213f5b29c" containerID="9455bb9efe86b293afe78151a548b6217c1cb9d6a25312305d43b6a765550c7e" exitCode=60 Nov 01 06:30:02.888350 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:02.887978 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v" event={"ID":"03e6f2ec-efdd-4594-98f0-6ec213f5b29c","Type":"ContainerDied","Data":"9455bb9efe86b293afe78151a548b6217c1cb9d6a25312305d43b6a765550c7e"} Nov 01 06:30:02.888350 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:02.888251 4385 scope.go:122] "RemoveContainer" containerID="9455bb9efe86b293afe78151a548b6217c1cb9d6a25312305d43b6a765550c7e" Nov 01 06:30:02.890459 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:02.890433 4385 generic.go:356] "Generic (PLEG): container finished" podID="6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7" containerID="2451ca7988bc7609cc9dbd945b71f43cf37d6f6694d46640b14d46a3bbbc07cd" exitCode=60 Nov 01 06:30:02.890582 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:02.890501 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" event={"ID":"6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7","Type":"ContainerDied","Data":"2451ca7988bc7609cc9dbd945b71f43cf37d6f6694d46640b14d46a3bbbc07cd"} Nov 01 06:30:02.890728 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:02.890712 4385 scope.go:122] "RemoveContainer" containerID="2451ca7988bc7609cc9dbd945b71f43cf37d6f6694d46640b14d46a3bbbc07cd" Nov 01 06:30:02.892517 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:02.892492 4385 generic.go:356] "Generic (PLEG): container finished" podID="32527ca5-6b0f-42c3-a007-da447e507f15" containerID="4f3073f7d72bb633d57e42a8bdf30e5fa3d98d9e370090ba9ca48cc52a228158" exitCode=60 Nov 01 06:30:02.892685 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:02.892517 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" event={"ID":"32527ca5-6b0f-42c3-a007-da447e507f15","Type":"ContainerDied","Data":"4f3073f7d72bb633d57e42a8bdf30e5fa3d98d9e370090ba9ca48cc52a228158"} Nov 01 06:30:02.892763 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:02.892749 4385 scope.go:122] "RemoveContainer" containerID="4f3073f7d72bb633d57e42a8bdf30e5fa3d98d9e370090ba9ca48cc52a228158" Nov 01 06:30:02.894287 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:02.894267 4385 generic.go:356] "Generic (PLEG): container finished" podID="dc17776b-6027-4263-b311-5b6c52ce8fc2" containerID="990ad84e19afa7da69fe3c521c99176d6567a356abe77a132c79963583f12633" exitCode=60 Nov 01 06:30:02.894407 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:02.894336 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" event={"ID":"dc17776b-6027-4263-b311-5b6c52ce8fc2","Type":"ContainerDied","Data":"990ad84e19afa7da69fe3c521c99176d6567a356abe77a132c79963583f12633"} Nov 01 06:30:02.894543 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:02.894524 4385 scope.go:122] "RemoveContainer" containerID="990ad84e19afa7da69fe3c521c99176d6567a356abe77a132c79963583f12633" Nov 01 06:30:03.081215 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.081186 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448" Nov 01 06:30:03.159254 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.159178 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dc828d8-18f2-4c17-935c-87572fa0001f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc828d8-18f2-4c17-935c-87572fa0001f-maas-api-openshift-ca\") pod \"6dc828d8-18f2-4c17-935c-87572fa0001f\" (UID: \"6dc828d8-18f2-4c17-935c-87572fa0001f\") " Nov 01 06:30:03.159254 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.159248 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dc828d8-18f2-4c17-935c-87572fa0001f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc828d8-18f2-4c17-935c-87572fa0001f-maas-api-xks-ca\") pod \"6dc828d8-18f2-4c17-935c-87572fa0001f\" (UID: \"6dc828d8-18f2-4c17-935c-87572fa0001f\") " Nov 01 06:30:03.159432 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.159365 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dc828d8-18f2-4c17-935c-87572fa0001f-kube-api-access-rpb2k\" (UniqueName: \"kubernetes.io/projected/6dc828d8-18f2-4c17-935c-87572fa0001f-kube-api-access-rpb2k\") pod \"6dc828d8-18f2-4c17-935c-87572fa0001f\" (UID: \"6dc828d8-18f2-4c17-935c-87572fa0001f\") " Nov 01 06:30:03.159547 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.159526 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc828d8-18f2-4c17-935c-87572fa0001f-maas-api-openshift-ca" pod "6dc828d8-18f2-4c17-935c-87572fa0001f" (UID: "6dc828d8-18f2-4c17-935c-87572fa0001f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:30:03.159984 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.159888 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc828d8-18f2-4c17-935c-87572fa0001f-maas-api-openshift-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:03.173027 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.173000 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dc828d8-18f2-4c17-935c-87572fa0001f-maas-api-xks-ca" pod "6dc828d8-18f2-4c17-935c-87572fa0001f" (UID: "6dc828d8-18f2-4c17-935c-87572fa0001f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:30:03.173197 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.173140 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc828d8-18f2-4c17-935c-87572fa0001f-kube-api-access-rpb2k" pod "6dc828d8-18f2-4c17-935c-87572fa0001f" (UID: "6dc828d8-18f2-4c17-935c-87572fa0001f"). InnerVolumeSpecName "kube-api-access-rpb2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:30:03.218506 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.218482 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f5d8a2a-zw9mn"] Nov 01 06:30:03.219036 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.219023 4385 cpu_manager.go:414] "containerMap: removing container" podUID="6dc828d8-18f2-4c17-935c-87572fa0001f" containerName="revoke-keys" Nov 01 06:30:03.219083 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.219037 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc828d8-18f2-4c17-935c-87572fa0001f" containerName="revoke-keys" Nov 01 06:30:03.219136 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.219116 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc828d8-18f2-4c17-935c-87572fa0001f" containerName="revoke-keys" Nov 01 06:30:03.222596 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.222583 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5d8a2a-zw9mn" Nov 01 06:30:03.261172 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.261151 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc828d8-18f2-4c17-935c-87572fa0001f-maas-api-xks-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:03.261288 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.261176 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpb2k\" (UniqueName: \"kubernetes.io/projected/6dc828d8-18f2-4c17-935c-87572fa0001f-kube-api-access-rpb2k\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:03.280963 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.280940 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f5d8a2a-zw9mn"] Nov 01 06:30:03.362819 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.362788 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jndhx\" (UniqueName: \"kubernetes.io/projected/ecb3a967-06bc-4169-b359-64ecf120e4e0-kube-api-access-jndhx\") pod \"maas-api-revoke-sub-2f5d8a2a-zw9mn\" (UID: \"ecb3a967-06bc-4169-b359-64ecf120e4e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5d8a2a-zw9mn" Nov 01 06:30:03.363026 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.362859 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb3a967-06bc-4169-b359-64ecf120e4e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f5d8a2a-zw9mn\" (UID: \"ecb3a967-06bc-4169-b359-64ecf120e4e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5d8a2a-zw9mn" Nov 01 06:30:03.363026 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.362931 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb3a967-06bc-4169-b359-64ecf120e4e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f5d8a2a-zw9mn\" (UID: \"ecb3a967-06bc-4169-b359-64ecf120e4e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5d8a2a-zw9mn" Nov 01 06:30:03.464793 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.464712 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jndhx\" (UniqueName: \"kubernetes.io/projected/ecb3a967-06bc-4169-b359-64ecf120e4e0-kube-api-access-jndhx\") pod \"maas-api-revoke-sub-2f5d8a2a-zw9mn\" (UID: \"ecb3a967-06bc-4169-b359-64ecf120e4e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5d8a2a-zw9mn" Nov 01 06:30:03.464793 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.464775 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb3a967-06bc-4169-b359-64ecf120e4e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f5d8a2a-zw9mn\" (UID: \"ecb3a967-06bc-4169-b359-64ecf120e4e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5d8a2a-zw9mn" Nov 01 06:30:03.464991 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.464804 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb3a967-06bc-4169-b359-64ecf120e4e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f5d8a2a-zw9mn\" (UID: \"ecb3a967-06bc-4169-b359-64ecf120e4e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5d8a2a-zw9mn" Nov 01 06:30:03.465663 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.465589 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb3a967-06bc-4169-b359-64ecf120e4e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f5d8a2a-zw9mn\" (UID: \"ecb3a967-06bc-4169-b359-64ecf120e4e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5d8a2a-zw9mn" Nov 01 06:30:03.467604 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.467576 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb3a967-06bc-4169-b359-64ecf120e4e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f5d8a2a-zw9mn\" (UID: \"ecb3a967-06bc-4169-b359-64ecf120e4e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5d8a2a-zw9mn" Nov 01 06:30:03.521400 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.521368 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jndhx\" (UniqueName: \"kubernetes.io/projected/ecb3a967-06bc-4169-b359-64ecf120e4e0-kube-api-access-jndhx\") pod \"maas-api-revoke-sub-2f5d8a2a-zw9mn\" (UID: \"ecb3a967-06bc-4169-b359-64ecf120e4e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5d8a2a-zw9mn" Nov 01 06:30:03.535300 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.535279 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5d8a2a-zw9mn" Nov 01 06:30:03.907209 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.907174 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" event={"ID":"dc17776b-6027-4263-b311-5b6c52ce8fc2","Type":"ContainerStarted","Data":"f7d5ad4a7902685064380b8d0a94a09246d7b09b89873563459658e7aa9425a0"} Nov 01 06:30:03.909339 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.909307 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448" event={"ID":"6dc828d8-18f2-4c17-935c-87572fa0001f","Type":"ContainerDied","Data":"d42b9e0aba40767c1dc44a44244bc282482b859390d387113ab8c08dbdc75a93"} Nov 01 06:30:03.909489 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.909362 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d42b9e0aba40767c1dc44a44244bc282482b859390d387113ab8c08dbdc75a93" Nov 01 06:30:03.909489 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.909422 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448" Nov 01 06:30:03.915635 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.915602 4385 generic.go:356] "Generic (PLEG): container finished" podID="03e6f2ec-efdd-4594-98f0-6ec213f5b29c" containerID="7360f9f6274d978086c5b4bd7fcbd74f8e4e3640793901edb9b0cc37f6c702bd" exitCode=60 Nov 01 06:30:03.916019 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.915767 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v" event={"ID":"03e6f2ec-efdd-4594-98f0-6ec213f5b29c","Type":"ContainerDied","Data":"7360f9f6274d978086c5b4bd7fcbd74f8e4e3640793901edb9b0cc37f6c702bd"} Nov 01 06:30:03.916019 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.915896 4385 scope.go:122] "RemoveContainer" containerID="9455bb9efe86b293afe78151a548b6217c1cb9d6a25312305d43b6a765550c7e" Nov 01 06:30:03.916019 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.915952 4385 scope.go:122] "RemoveContainer" containerID="7360f9f6274d978086c5b4bd7fcbd74f8e4e3640793901edb9b0cc37f6c702bd" Nov 01 06:30:03.916275 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:30:03.916246 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891910-xw57v_odh-ai-gateway-infra(03e6f2ec-efdd-4594-98f0-6ec213f5b29c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v" podUID="03e6f2ec-efdd-4594-98f0-6ec213f5b29c" Nov 01 06:30:03.927054 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.927010 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" event={"ID":"6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7","Type":"ContainerStarted","Data":"fd8dcb2e58d354f50750ba7f7b1c8739b269a21626c81498c795140e15c3ca88"} Nov 01 06:30:03.929658 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.929639 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" event={"ID":"32527ca5-6b0f-42c3-a007-da447e507f15","Type":"ContainerStarted","Data":"a8a21560c786a46206697d54bcd9876296907c554def6a636589fda5dfb9d054"} Nov 01 06:30:03.933117 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:30:03.933094 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb3a967_06bc_4169_b359_64ecf120e4e0.slice/crio-ad73a3ec750ab44030f0adc3f1e1e4bf7d9315f8eb42856467c24c29e3525c78 WatchSource:0}: Error finding container ad73a3ec750ab44030f0adc3f1e1e4bf7d9315f8eb42856467c24c29e3525c78: Status 404 returned error can't find the container with id ad73a3ec750ab44030f0adc3f1e1e4bf7d9315f8eb42856467c24c29e3525c78 Nov 01 06:30:03.933201 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:03.933130 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f5d8a2a-zw9mn"] Nov 01 06:30:04.941914 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:04.941888 4385 scope.go:122] "RemoveContainer" containerID="7360f9f6274d978086c5b4bd7fcbd74f8e4e3640793901edb9b0cc37f6c702bd" Nov 01 06:30:04.942285 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:30:04.942221 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891910-xw57v_odh-ai-gateway-infra(03e6f2ec-efdd-4594-98f0-6ec213f5b29c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v" podUID="03e6f2ec-efdd-4594-98f0-6ec213f5b29c" Nov 01 06:30:04.943633 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:04.943609 4385 generic.go:356] "Generic (PLEG): container finished" podID="6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7" containerID="fd8dcb2e58d354f50750ba7f7b1c8739b269a21626c81498c795140e15c3ca88" exitCode=60 Nov 01 06:30:04.943728 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:04.943684 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" event={"ID":"6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7","Type":"ContainerDied","Data":"fd8dcb2e58d354f50750ba7f7b1c8739b269a21626c81498c795140e15c3ca88"} Nov 01 06:30:04.943779 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:04.943738 4385 scope.go:122] "RemoveContainer" containerID="2451ca7988bc7609cc9dbd945b71f43cf37d6f6694d46640b14d46a3bbbc07cd" Nov 01 06:30:04.943890 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:04.943877 4385 scope.go:122] "RemoveContainer" containerID="fd8dcb2e58d354f50750ba7f7b1c8739b269a21626c81498c795140e15c3ca88" Nov 01 06:30:04.944121 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:30:04.944105 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc_odh-ai-gateway-infra(6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" podUID="6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7" Nov 01 06:30:04.945810 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:04.945792 4385 generic.go:356] "Generic (PLEG): container finished" podID="32527ca5-6b0f-42c3-a007-da447e507f15" containerID="a8a21560c786a46206697d54bcd9876296907c554def6a636589fda5dfb9d054" exitCode=60 Nov 01 06:30:04.945921 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:04.945858 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" event={"ID":"32527ca5-6b0f-42c3-a007-da447e507f15","Type":"ContainerDied","Data":"a8a21560c786a46206697d54bcd9876296907c554def6a636589fda5dfb9d054"} Nov 01 06:30:04.946080 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:04.946067 4385 scope.go:122] "RemoveContainer" containerID="a8a21560c786a46206697d54bcd9876296907c554def6a636589fda5dfb9d054" Nov 01 06:30:04.946293 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:30:04.946274 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786_odh-ai-gateway-infra(32527ca5-6b0f-42c3-a007-da447e507f15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" podUID="32527ca5-6b0f-42c3-a007-da447e507f15" Nov 01 06:30:04.947656 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:04.947631 4385 generic.go:356] "Generic (PLEG): container finished" podID="dc17776b-6027-4263-b311-5b6c52ce8fc2" containerID="f7d5ad4a7902685064380b8d0a94a09246d7b09b89873563459658e7aa9425a0" exitCode=60 Nov 01 06:30:04.947736 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:04.947652 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" event={"ID":"dc17776b-6027-4263-b311-5b6c52ce8fc2","Type":"ContainerDied","Data":"f7d5ad4a7902685064380b8d0a94a09246d7b09b89873563459658e7aa9425a0"} Nov 01 06:30:04.947904 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:04.947888 4385 scope.go:122] "RemoveContainer" containerID="f7d5ad4a7902685064380b8d0a94a09246d7b09b89873563459658e7aa9425a0" Nov 01 06:30:04.948106 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:30:04.948085 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr_odh-ai-gateway-infra(dc17776b-6027-4263-b311-5b6c52ce8fc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" podUID="dc17776b-6027-4263-b311-5b6c52ce8fc2" Nov 01 06:30:04.949528 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:04.949506 4385 generic.go:356] "Generic (PLEG): container finished" podID="ecb3a967-06bc-4169-b359-64ecf120e4e0" containerID="49632d496c0f05d898e1ddc0ef8ea9c9e7f366b073fdf6606c95cec92bb46019" exitCode=0 Nov 01 06:30:04.949610 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:04.949584 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5d8a2a-zw9mn" event={"ID":"ecb3a967-06bc-4169-b359-64ecf120e4e0","Type":"ContainerDied","Data":"49632d496c0f05d898e1ddc0ef8ea9c9e7f366b073fdf6606c95cec92bb46019"} Nov 01 06:30:04.949655 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:04.949621 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5d8a2a-zw9mn" event={"ID":"ecb3a967-06bc-4169-b359-64ecf120e4e0","Type":"ContainerStarted","Data":"ad73a3ec750ab44030f0adc3f1e1e4bf7d9315f8eb42856467c24c29e3525c78"} Nov 01 06:30:04.957126 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:04.957109 4385 scope.go:122] "RemoveContainer" containerID="4f3073f7d72bb633d57e42a8bdf30e5fa3d98d9e370090ba9ca48cc52a228158" Nov 01 06:30:04.969870 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:04.969849 4385 scope.go:122] "RemoveContainer" containerID="990ad84e19afa7da69fe3c521c99176d6567a356abe77a132c79963583f12633" Nov 01 06:30:05.961069 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:05.961039 4385 scope.go:122] "RemoveContainer" containerID="fd8dcb2e58d354f50750ba7f7b1c8739b269a21626c81498c795140e15c3ca88" Nov 01 06:30:05.961614 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:30:05.961345 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc_odh-ai-gateway-infra(6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" podUID="6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7" Nov 01 06:30:05.962692 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:05.962662 4385 scope.go:122] "RemoveContainer" containerID="a8a21560c786a46206697d54bcd9876296907c554def6a636589fda5dfb9d054" Nov 01 06:30:05.962951 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:30:05.962931 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786_odh-ai-gateway-infra(32527ca5-6b0f-42c3-a007-da447e507f15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" podUID="32527ca5-6b0f-42c3-a007-da447e507f15" Nov 01 06:30:05.964150 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:05.964132 4385 scope.go:122] "RemoveContainer" containerID="f7d5ad4a7902685064380b8d0a94a09246d7b09b89873563459658e7aa9425a0" Nov 01 06:30:05.964613 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:30:05.964341 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr_odh-ai-gateway-infra(dc17776b-6027-4263-b311-5b6c52ce8fc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" podUID="dc17776b-6027-4263-b311-5b6c52ce8fc2" Nov 01 06:30:06.115795 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:06.115768 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5d8a2a-zw9mn" Nov 01 06:30:06.135205 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:06.135173 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:06.223532 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:06.223454 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb3a967-06bc-4169-b359-64ecf120e4e0-kube-api-access-jndhx\" (UniqueName: \"kubernetes.io/projected/ecb3a967-06bc-4169-b359-64ecf120e4e0-kube-api-access-jndhx\") pod \"ecb3a967-06bc-4169-b359-64ecf120e4e0\" (UID: \"ecb3a967-06bc-4169-b359-64ecf120e4e0\") " Nov 01 06:30:06.223680 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:06.223559 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecb3a967-06bc-4169-b359-64ecf120e4e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb3a967-06bc-4169-b359-64ecf120e4e0-maas-api-openshift-ca\") pod \"ecb3a967-06bc-4169-b359-64ecf120e4e0\" (UID: \"ecb3a967-06bc-4169-b359-64ecf120e4e0\") " Nov 01 06:30:06.223680 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:06.223623 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecb3a967-06bc-4169-b359-64ecf120e4e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb3a967-06bc-4169-b359-64ecf120e4e0-maas-api-xks-ca\") pod \"ecb3a967-06bc-4169-b359-64ecf120e4e0\" (UID: \"ecb3a967-06bc-4169-b359-64ecf120e4e0\") " Nov 01 06:30:06.223955 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:06.223931 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb3a967-06bc-4169-b359-64ecf120e4e0-maas-api-openshift-ca" pod "ecb3a967-06bc-4169-b359-64ecf120e4e0" (UID: "ecb3a967-06bc-4169-b359-64ecf120e4e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:30:06.225932 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:06.225897 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb3a967-06bc-4169-b359-64ecf120e4e0-kube-api-access-jndhx" pod "ecb3a967-06bc-4169-b359-64ecf120e4e0" (UID: "ecb3a967-06bc-4169-b359-64ecf120e4e0"). InnerVolumeSpecName "kube-api-access-jndhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:30:06.226056 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:06.225943 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb3a967-06bc-4169-b359-64ecf120e4e0-maas-api-xks-ca" pod "ecb3a967-06bc-4169-b359-64ecf120e4e0" (UID: "ecb3a967-06bc-4169-b359-64ecf120e4e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:30:06.325238 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:06.325210 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb3a967-06bc-4169-b359-64ecf120e4e0-maas-api-openshift-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:06.325238 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:06.325236 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb3a967-06bc-4169-b359-64ecf120e4e0-maas-api-xks-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:06.325429 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:06.325246 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jndhx\" (UniqueName: \"kubernetes.io/projected/ecb3a967-06bc-4169-b359-64ecf120e4e0-kube-api-access-jndhx\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:06.794261 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:30:06.793740 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:30:06.989857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:06.985649 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5d8a2a-zw9mn" event={"ID":"ecb3a967-06bc-4169-b359-64ecf120e4e0","Type":"ContainerDied","Data":"ad73a3ec750ab44030f0adc3f1e1e4bf7d9315f8eb42856467c24c29e3525c78"} Nov 01 06:30:06.989857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:06.985755 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad73a3ec750ab44030f0adc3f1e1e4bf7d9315f8eb42856467c24c29e3525c78" Nov 01 06:30:06.989857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:06.985850 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5d8a2a-zw9mn" Nov 01 06:30:09.239676 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.239645 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6"] Nov 01 06:30:09.240204 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.240189 4385 cpu_manager.go:414] "containerMap: removing container" podUID="ecb3a967-06bc-4169-b359-64ecf120e4e0" containerName="revoke-keys" Nov 01 06:30:09.240248 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.240206 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecb3a967-06bc-4169-b359-64ecf120e4e0" containerName="revoke-keys" Nov 01 06:30:09.240290 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.240285 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecb3a967-06bc-4169-b359-64ecf120e4e0" containerName="revoke-keys" Nov 01 06:30:09.243593 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.243578 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:09.279792 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.279755 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-dshm\") pod \"ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:09.279985 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.279946 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-tmp-dir\") pod \"ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:09.280054 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.280002 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-model-cache\") pod \"ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:09.280107 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.280048 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs2zk\" (UniqueName: \"kubernetes.io/projected/ade9c35c-3890-4fca-844a-7082358cbcf6-kube-api-access-xs2zk\") pod \"ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:09.280107 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.280078 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-home\") pod \"ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:09.280198 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.280124 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade9c35c-3890-4fca-844a-7082358cbcf6-tls-certs\") pod \"ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:09.300396 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.300355 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6"] Nov 01 06:30:09.336625 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.336573 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:09.345327 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.345298 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-bd9d24\"/\"openshift-service-ca.crt\"" Nov 01 06:30:09.345327 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.345319 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-bd9d24\"/\"kube-root-ca.crt\"" Nov 01 06:30:09.345327 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.345299 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-bd9d24\"/\"default-dockercfg-7x2d9\"" Nov 01 06:30:09.345568 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.345398 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-bd9d24\"/\"ipp-route-bd9d24-kserve-self-signed-certs\"" Nov 01 06:30:09.390442 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.388356 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-model-cache\") pod \"ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:09.390442 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.388416 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs2zk\" (UniqueName: \"kubernetes.io/projected/ade9c35c-3890-4fca-844a-7082358cbcf6-kube-api-access-xs2zk\") pod \"ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:09.390442 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.388450 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-home\") pod \"ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:09.390442 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.388497 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade9c35c-3890-4fca-844a-7082358cbcf6-tls-certs\") pod \"ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:09.390442 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.388618 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-dshm\") pod \"ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:09.390442 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.388696 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-tmp-dir\") pod \"ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:09.390442 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.389168 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-tmp-dir\") pod \"ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:09.390442 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.389398 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-model-cache\") pod \"ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:09.390442 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.389944 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-home\") pod \"ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:09.399498 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.399471 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-dshm\") pod \"ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:09.400637 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.400571 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade9c35c-3890-4fca-844a-7082358cbcf6-tls-certs\") pod \"ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:09.445791 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.445753 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs2zk\" (UniqueName: \"kubernetes.io/projected/ade9c35c-3890-4fca-844a-7082358cbcf6-kube-api-access-xs2zk\") pod \"ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:09.556456 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.556382 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:09.961892 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:30:09.961852 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podade9c35c_3890_4fca_844a_7082358cbcf6.slice/crio-efc0c707221ab59e9810646e64b9fe1606c02df955e82287dd1a7eda3505952c WatchSource:0}: Error finding container efc0c707221ab59e9810646e64b9fe1606c02df955e82287dd1a7eda3505952c: Status 404 returned error can't find the container with id efc0c707221ab59e9810646e64b9fe1606c02df955e82287dd1a7eda3505952c Nov 01 06:30:09.962568 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:09.962546 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6"] Nov 01 06:30:10.012440 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:10.012410 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" event={"ID":"ade9c35c-3890-4fca-844a-7082358cbcf6","Type":"ContainerStarted","Data":"efc0c707221ab59e9810646e64b9fe1606c02df955e82287dd1a7eda3505952c"} Nov 01 06:30:11.030150 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:11.030111 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" event={"ID":"ade9c35c-3890-4fca-844a-7082358cbcf6","Type":"ContainerStarted","Data":"610a8725d6b24651479631a63279ed199d73056a460ccaf9df05263429e150dd"} Nov 01 06:30:11.144490 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:11.144436 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" podStartSLOduration=2.144422053 podStartE2EDuration="2.144422053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:11.142025926 +0000 UTC m=+1220.905825769" watchObservedRunningTime="2026-11-01 06:30:11.144422053 +0000 UTC m=+1220.908221910" Nov 01 06:30:11.446370 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:11.446320 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:11.557129 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:11.557096 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:11.649852 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:11.649800 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:12.641607 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:12.641557 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:13.242407 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:13.242374 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:13.390859 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:13.390297 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:13.968882 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:13.968845 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e78798ce-fkfwc"] Nov 01 06:30:13.974052 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:13.974028 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78798ce-fkfwc" Nov 01 06:30:14.032327 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.032299 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e78798ce-fkfwc"] Nov 01 06:30:14.050240 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.050211 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r628\" (UniqueName: \"kubernetes.io/projected/3fa42472-eaf4-4a8a-9b47-64a708c379c5-kube-api-access-2r628\") pod \"maas-api-revoke-sub-e78798ce-fkfwc\" (UID: \"3fa42472-eaf4-4a8a-9b47-64a708c379c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78798ce-fkfwc" Nov 01 06:30:14.050395 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.050262 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa42472-eaf4-4a8a-9b47-64a708c379c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e78798ce-fkfwc\" (UID: \"3fa42472-eaf4-4a8a-9b47-64a708c379c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78798ce-fkfwc" Nov 01 06:30:14.050482 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.050421 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa42472-eaf4-4a8a-9b47-64a708c379c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e78798ce-fkfwc\" (UID: \"3fa42472-eaf4-4a8a-9b47-64a708c379c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78798ce-fkfwc" Nov 01 06:30:14.075656 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.075628 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:30:14.151452 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.151422 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r628\" (UniqueName: \"kubernetes.io/projected/3fa42472-eaf4-4a8a-9b47-64a708c379c5-kube-api-access-2r628\") pod \"maas-api-revoke-sub-e78798ce-fkfwc\" (UID: \"3fa42472-eaf4-4a8a-9b47-64a708c379c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78798ce-fkfwc" Nov 01 06:30:14.151617 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.151462 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa42472-eaf4-4a8a-9b47-64a708c379c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e78798ce-fkfwc\" (UID: \"3fa42472-eaf4-4a8a-9b47-64a708c379c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78798ce-fkfwc" Nov 01 06:30:14.151617 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.151532 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa42472-eaf4-4a8a-9b47-64a708c379c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e78798ce-fkfwc\" (UID: \"3fa42472-eaf4-4a8a-9b47-64a708c379c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78798ce-fkfwc" Nov 01 06:30:14.152308 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.152250 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa42472-eaf4-4a8a-9b47-64a708c379c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e78798ce-fkfwc\" (UID: \"3fa42472-eaf4-4a8a-9b47-64a708c379c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78798ce-fkfwc" Nov 01 06:30:14.157295 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.156758 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa42472-eaf4-4a8a-9b47-64a708c379c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e78798ce-fkfwc\" (UID: \"3fa42472-eaf4-4a8a-9b47-64a708c379c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78798ce-fkfwc" Nov 01 06:30:14.207906 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.207874 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r628\" (UniqueName: \"kubernetes.io/projected/3fa42472-eaf4-4a8a-9b47-64a708c379c5-kube-api-access-2r628\") pod \"maas-api-revoke-sub-e78798ce-fkfwc\" (UID: \"3fa42472-eaf4-4a8a-9b47-64a708c379c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78798ce-fkfwc" Nov 01 06:30:14.288960 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.288891 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78798ce-fkfwc" Nov 01 06:30:14.679114 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:30:14.679068 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa42472_eaf4_4a8a_9b47_64a708c379c5.slice/crio-edd0b595bdd8a04aca90650c6250c829e14ab95aab61843d1b2fc9b46f540a5b WatchSource:0}: Error finding container edd0b595bdd8a04aca90650c6250c829e14ab95aab61843d1b2fc9b46f540a5b: Status 404 returned error can't find the container with id edd0b595bdd8a04aca90650c6250c829e14ab95aab61843d1b2fc9b46f540a5b Nov 01 06:30:14.682878 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.682856 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e78798ce-fkfwc"] Nov 01 06:30:14.798294 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.798256 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77"] Nov 01 06:30:14.803574 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.803553 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77" Nov 01 06:30:14.858188 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.858158 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77"] Nov 01 06:30:14.860069 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.859883 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f84219e-7934-4975-a98f-1d8b04edbc98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b78c0aae-tmq77\" (UID: \"5f84219e-7934-4975-a98f-1d8b04edbc98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77" Nov 01 06:30:14.860069 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.859984 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9sz6\" (UniqueName: \"kubernetes.io/projected/5f84219e-7934-4975-a98f-1d8b04edbc98-kube-api-access-n9sz6\") pod \"maas-api-revoke-sub-b78c0aae-tmq77\" (UID: \"5f84219e-7934-4975-a98f-1d8b04edbc98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77" Nov 01 06:30:14.860285 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.860127 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f84219e-7934-4975-a98f-1d8b04edbc98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b78c0aae-tmq77\" (UID: \"5f84219e-7934-4975-a98f-1d8b04edbc98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77" Nov 01 06:30:14.961814 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.961725 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9sz6\" (UniqueName: \"kubernetes.io/projected/5f84219e-7934-4975-a98f-1d8b04edbc98-kube-api-access-n9sz6\") pod \"maas-api-revoke-sub-b78c0aae-tmq77\" (UID: \"5f84219e-7934-4975-a98f-1d8b04edbc98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77" Nov 01 06:30:14.962006 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.961885 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f84219e-7934-4975-a98f-1d8b04edbc98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b78c0aae-tmq77\" (UID: \"5f84219e-7934-4975-a98f-1d8b04edbc98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77" Nov 01 06:30:14.962006 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.961995 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f84219e-7934-4975-a98f-1d8b04edbc98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b78c0aae-tmq77\" (UID: \"5f84219e-7934-4975-a98f-1d8b04edbc98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77" Nov 01 06:30:14.962727 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.962699 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f84219e-7934-4975-a98f-1d8b04edbc98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b78c0aae-tmq77\" (UID: \"5f84219e-7934-4975-a98f-1d8b04edbc98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77" Nov 01 06:30:14.964443 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:14.964419 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f84219e-7934-4975-a98f-1d8b04edbc98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b78c0aae-tmq77\" (UID: \"5f84219e-7934-4975-a98f-1d8b04edbc98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77" Nov 01 06:30:15.018234 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:15.018203 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9sz6\" (UniqueName: \"kubernetes.io/projected/5f84219e-7934-4975-a98f-1d8b04edbc98-kube-api-access-n9sz6\") pod \"maas-api-revoke-sub-b78c0aae-tmq77\" (UID: \"5f84219e-7934-4975-a98f-1d8b04edbc98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77" Nov 01 06:30:15.064454 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:15.064419 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78798ce-fkfwc" event={"ID":"3fa42472-eaf4-4a8a-9b47-64a708c379c5","Type":"ContainerStarted","Data":"edd0b595bdd8a04aca90650c6250c829e14ab95aab61843d1b2fc9b46f540a5b"} Nov 01 06:30:15.126315 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:15.126285 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77" Nov 01 06:30:15.377363 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:15.377332 4385 scope.go:122] "RemoveContainer" containerID="a8a21560c786a46206697d54bcd9876296907c554def6a636589fda5dfb9d054" Nov 01 06:30:15.377684 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:15.377665 4385 scope.go:122] "RemoveContainer" containerID="f7d5ad4a7902685064380b8d0a94a09246d7b09b89873563459658e7aa9425a0" Nov 01 06:30:15.509632 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:15.509605 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77"] Nov 01 06:30:16.075851 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:16.075806 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" event={"ID":"32527ca5-6b0f-42c3-a007-da447e507f15","Type":"ContainerStarted","Data":"1386e05228cdd180f5e7d7dd98a758bd1b8949e87f330b67b102ef69bc44ebeb"} Nov 01 06:30:16.077892 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:16.077865 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" event={"ID":"dc17776b-6027-4263-b311-5b6c52ce8fc2","Type":"ContainerStarted","Data":"898028e5ecc1c190a3d9eddcb1537aa6cfec706cf314ec46fdb9ba734fb10d3a"} Nov 01 06:30:16.079507 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:16.079486 4385 generic.go:356] "Generic (PLEG): container finished" podID="3fa42472-eaf4-4a8a-9b47-64a708c379c5" containerID="626928765bcc8762b092c74a0998a59778f214b113ab1117cb937b730c50aba1" exitCode=0 Nov 01 06:30:16.079592 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:16.079527 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78798ce-fkfwc" event={"ID":"3fa42472-eaf4-4a8a-9b47-64a708c379c5","Type":"ContainerDied","Data":"626928765bcc8762b092c74a0998a59778f214b113ab1117cb937b730c50aba1"} Nov 01 06:30:16.081227 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:16.081206 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77" event={"ID":"5f84219e-7934-4975-a98f-1d8b04edbc98","Type":"ContainerStarted","Data":"2fb15bef1615a710c85a6f65be1da18aef15d7762d45d3d091bce72b9f6735b5"} Nov 01 06:30:16.081307 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:16.081252 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77" event={"ID":"5f84219e-7934-4975-a98f-1d8b04edbc98","Type":"ContainerStarted","Data":"36f0ab2567f8a48410d7cbe825fd15294c95d37e710aa5e02ec5de82b13d9247"} Nov 01 06:30:16.305618 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:16.305551 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77" podStartSLOduration=2.305532456 podStartE2EDuration="2.305532456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:16.300039958 +0000 UTC m=+1226.063839815" watchObservedRunningTime="2026-11-01 06:30:16.305532456 +0000 UTC m=+1226.069332314" Nov 01 06:30:17.045667 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.045625 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr"] Nov 01 06:30:17.090121 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.090087 4385 generic.go:356] "Generic (PLEG): container finished" podID="32527ca5-6b0f-42c3-a007-da447e507f15" containerID="1386e05228cdd180f5e7d7dd98a758bd1b8949e87f330b67b102ef69bc44ebeb" exitCode=60 Nov 01 06:30:17.090615 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.090170 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" event={"ID":"32527ca5-6b0f-42c3-a007-da447e507f15","Type":"ContainerDied","Data":"1386e05228cdd180f5e7d7dd98a758bd1b8949e87f330b67b102ef69bc44ebeb"} Nov 01 06:30:17.090615 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.090247 4385 scope.go:122] "RemoveContainer" containerID="a8a21560c786a46206697d54bcd9876296907c554def6a636589fda5dfb9d054" Nov 01 06:30:17.090716 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.090680 4385 scope.go:122] "RemoveContainer" containerID="1386e05228cdd180f5e7d7dd98a758bd1b8949e87f330b67b102ef69bc44ebeb" Nov 01 06:30:17.091002 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:30:17.090973 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786_odh-ai-gateway-infra(32527ca5-6b0f-42c3-a007-da447e507f15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" podUID="32527ca5-6b0f-42c3-a007-da447e507f15" Nov 01 06:30:17.092643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.092621 4385 generic.go:356] "Generic (PLEG): container finished" podID="dc17776b-6027-4263-b311-5b6c52ce8fc2" containerID="898028e5ecc1c190a3d9eddcb1537aa6cfec706cf314ec46fdb9ba734fb10d3a" exitCode=60 Nov 01 06:30:17.092773 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.092696 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" event={"ID":"dc17776b-6027-4263-b311-5b6c52ce8fc2","Type":"ContainerDied","Data":"898028e5ecc1c190a3d9eddcb1537aa6cfec706cf314ec46fdb9ba734fb10d3a"} Nov 01 06:30:17.093065 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.093046 4385 scope.go:122] "RemoveContainer" containerID="898028e5ecc1c190a3d9eddcb1537aa6cfec706cf314ec46fdb9ba734fb10d3a" Nov 01 06:30:17.093268 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:30:17.093249 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr_odh-ai-gateway-infra(dc17776b-6027-4263-b311-5b6c52ce8fc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" podUID="dc17776b-6027-4263-b311-5b6c52ce8fc2" Nov 01 06:30:17.094478 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.094455 4385 generic.go:356] "Generic (PLEG): container finished" podID="5f84219e-7934-4975-a98f-1d8b04edbc98" containerID="2fb15bef1615a710c85a6f65be1da18aef15d7762d45d3d091bce72b9f6735b5" exitCode=0 Nov 01 06:30:17.094585 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.094483 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77" event={"ID":"5f84219e-7934-4975-a98f-1d8b04edbc98","Type":"ContainerDied","Data":"2fb15bef1615a710c85a6f65be1da18aef15d7762d45d3d091bce72b9f6735b5"} Nov 01 06:30:17.099872 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.096660 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a80c5c9-g72tr"] Nov 01 06:30:17.110611 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.110593 4385 scope.go:122] "RemoveContainer" containerID="f7d5ad4a7902685064380b8d0a94a09246d7b09b89873563459658e7aa9425a0" Nov 01 06:30:17.204436 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.204403 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786"] Nov 01 06:30:17.258763 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.258717 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78798ce-fkfwc" Nov 01 06:30:17.293393 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.293369 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa42472-eaf4-4a8a-9b47-64a708c379c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa42472-eaf4-4a8a-9b47-64a708c379c5-maas-api-openshift-ca\") pod \"3fa42472-eaf4-4a8a-9b47-64a708c379c5\" (UID: \"3fa42472-eaf4-4a8a-9b47-64a708c379c5\") " Nov 01 06:30:17.293519 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.293460 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa42472-eaf4-4a8a-9b47-64a708c379c5-kube-api-access-2r628\" (UniqueName: \"kubernetes.io/projected/3fa42472-eaf4-4a8a-9b47-64a708c379c5-kube-api-access-2r628\") pod \"3fa42472-eaf4-4a8a-9b47-64a708c379c5\" (UID: \"3fa42472-eaf4-4a8a-9b47-64a708c379c5\") " Nov 01 06:30:17.293557 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.293518 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fa42472-eaf4-4a8a-9b47-64a708c379c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa42472-eaf4-4a8a-9b47-64a708c379c5-maas-api-xks-ca\") pod \"3fa42472-eaf4-4a8a-9b47-64a708c379c5\" (UID: \"3fa42472-eaf4-4a8a-9b47-64a708c379c5\") " Nov 01 06:30:17.293676 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.293657 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa42472-eaf4-4a8a-9b47-64a708c379c5-maas-api-openshift-ca" pod "3fa42472-eaf4-4a8a-9b47-64a708c379c5" (UID: "3fa42472-eaf4-4a8a-9b47-64a708c379c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:30:17.293898 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.293879 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa42472-eaf4-4a8a-9b47-64a708c379c5-maas-api-openshift-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:17.295742 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.295689 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa42472-eaf4-4a8a-9b47-64a708c379c5-maas-api-xks-ca" pod "3fa42472-eaf4-4a8a-9b47-64a708c379c5" (UID: "3fa42472-eaf4-4a8a-9b47-64a708c379c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:30:17.295742 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.295730 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa42472-eaf4-4a8a-9b47-64a708c379c5-kube-api-access-2r628" pod "3fa42472-eaf4-4a8a-9b47-64a708c379c5" (UID: "3fa42472-eaf4-4a8a-9b47-64a708c379c5"). InnerVolumeSpecName "kube-api-access-2r628". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:30:17.377188 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.377165 4385 scope.go:122] "RemoveContainer" containerID="fd8dcb2e58d354f50750ba7f7b1c8739b269a21626c81498c795140e15c3ca88" Nov 01 06:30:17.377328 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.377299 4385 scope.go:122] "RemoveContainer" containerID="7360f9f6274d978086c5b4bd7fcbd74f8e4e3640793901edb9b0cc37f6c702bd" Nov 01 06:30:17.394555 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.394520 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r628\" (UniqueName: \"kubernetes.io/projected/3fa42472-eaf4-4a8a-9b47-64a708c379c5-kube-api-access-2r628\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:17.394555 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.394541 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa42472-eaf4-4a8a-9b47-64a708c379c5-maas-api-xks-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:17.535636 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.535605 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr"] Nov 01 06:30:17.663488 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.663443 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:17.834427 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:17.834341 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:18.006702 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.006661 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:18.103359 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.103274 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v" event={"ID":"03e6f2ec-efdd-4594-98f0-6ec213f5b29c","Type":"ContainerStarted","Data":"15ac541b4ea227cdc0acf47486265d85c419866cfca23a6b7375d7c7c457d8b5"} Nov 01 06:30:18.105254 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.105223 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" event={"ID":"6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7","Type":"ContainerStarted","Data":"d3a62d0094ff834c8f7a60af60cfeb2e4fff73a9ab84f545d5be275a840c302f"} Nov 01 06:30:18.110522 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.110504 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78798ce-fkfwc" Nov 01 06:30:18.110625 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.110538 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e78798ce-fkfwc" event={"ID":"3fa42472-eaf4-4a8a-9b47-64a708c379c5","Type":"ContainerDied","Data":"edd0b595bdd8a04aca90650c6250c829e14ab95aab61843d1b2fc9b46f540a5b"} Nov 01 06:30:18.110625 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.110591 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edd0b595bdd8a04aca90650c6250c829e14ab95aab61843d1b2fc9b46f540a5b" Nov 01 06:30:18.305890 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.301737 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77" Nov 01 06:30:18.329641 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.329620 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" Nov 01 06:30:18.337078 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.337050 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" Nov 01 06:30:18.385676 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.385650 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ba053c1-ca6c-4dbf-b96d-091354a3c245" path="/var/lib/kubelet/pods/0ba053c1-ca6c-4dbf-b96d-091354a3c245/volumes" Nov 01 06:30:18.405151 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.405129 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f84219e-7934-4975-a98f-1d8b04edbc98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f84219e-7934-4975-a98f-1d8b04edbc98-maas-api-xks-ca\") pod \"5f84219e-7934-4975-a98f-1d8b04edbc98\" (UID: \"5f84219e-7934-4975-a98f-1d8b04edbc98\") " Nov 01 06:30:18.405295 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.405173 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f84219e-7934-4975-a98f-1d8b04edbc98-kube-api-access-n9sz6\" (UniqueName: \"kubernetes.io/projected/5f84219e-7934-4975-a98f-1d8b04edbc98-kube-api-access-n9sz6\") pod \"5f84219e-7934-4975-a98f-1d8b04edbc98\" (UID: \"5f84219e-7934-4975-a98f-1d8b04edbc98\") " Nov 01 06:30:18.405295 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.405247 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f84219e-7934-4975-a98f-1d8b04edbc98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f84219e-7934-4975-a98f-1d8b04edbc98-maas-api-openshift-ca\") pod \"5f84219e-7934-4975-a98f-1d8b04edbc98\" (UID: \"5f84219e-7934-4975-a98f-1d8b04edbc98\") " Nov 01 06:30:18.405295 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.405276 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc17776b-6027-4263-b311-5b6c52ce8fc2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc17776b-6027-4263-b311-5b6c52ce8fc2-maas-api-service-ca\") pod \"dc17776b-6027-4263-b311-5b6c52ce8fc2\" (UID: \"dc17776b-6027-4263-b311-5b6c52ce8fc2\") " Nov 01 06:30:18.405490 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.405387 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32527ca5-6b0f-42c3-a007-da447e507f15-kube-api-access-s7ltf\" (UniqueName: \"kubernetes.io/projected/32527ca5-6b0f-42c3-a007-da447e507f15-kube-api-access-s7ltf\") pod \"32527ca5-6b0f-42c3-a007-da447e507f15\" (UID: \"32527ca5-6b0f-42c3-a007-da447e507f15\") " Nov 01 06:30:18.405557 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.405510 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32527ca5-6b0f-42c3-a007-da447e507f15-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32527ca5-6b0f-42c3-a007-da447e507f15-maas-api-service-ca\") pod \"32527ca5-6b0f-42c3-a007-da447e507f15\" (UID: \"32527ca5-6b0f-42c3-a007-da447e507f15\") " Nov 01 06:30:18.406067 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.405560 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc17776b-6027-4263-b311-5b6c52ce8fc2-kube-api-access-4clbk\" (UniqueName: \"kubernetes.io/projected/dc17776b-6027-4263-b311-5b6c52ce8fc2-kube-api-access-4clbk\") pod \"dc17776b-6027-4263-b311-5b6c52ce8fc2\" (UID: \"dc17776b-6027-4263-b311-5b6c52ce8fc2\") " Nov 01 06:30:18.406258 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.406147 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f84219e-7934-4975-a98f-1d8b04edbc98-maas-api-openshift-ca" pod "5f84219e-7934-4975-a98f-1d8b04edbc98" (UID: "5f84219e-7934-4975-a98f-1d8b04edbc98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:30:18.406630 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.406274 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc17776b-6027-4263-b311-5b6c52ce8fc2-maas-api-service-ca" pod "dc17776b-6027-4263-b311-5b6c52ce8fc2" (UID: "dc17776b-6027-4263-b311-5b6c52ce8fc2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:30:18.406630 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.406529 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32527ca5-6b0f-42c3-a007-da447e507f15-maas-api-service-ca" pod "32527ca5-6b0f-42c3-a007-da447e507f15" (UID: "32527ca5-6b0f-42c3-a007-da447e507f15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:30:18.408487 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.408435 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f84219e-7934-4975-a98f-1d8b04edbc98-maas-api-xks-ca" pod "5f84219e-7934-4975-a98f-1d8b04edbc98" (UID: "5f84219e-7934-4975-a98f-1d8b04edbc98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:30:18.408741 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.408608 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc17776b-6027-4263-b311-5b6c52ce8fc2-kube-api-access-4clbk" pod "dc17776b-6027-4263-b311-5b6c52ce8fc2" (UID: "dc17776b-6027-4263-b311-5b6c52ce8fc2"). InnerVolumeSpecName "kube-api-access-4clbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:30:18.408741 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.408705 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32527ca5-6b0f-42c3-a007-da447e507f15-kube-api-access-s7ltf" pod "32527ca5-6b0f-42c3-a007-da447e507f15" (UID: "32527ca5-6b0f-42c3-a007-da447e507f15"). InnerVolumeSpecName "kube-api-access-s7ltf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:30:18.409005 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.408979 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f84219e-7934-4975-a98f-1d8b04edbc98-kube-api-access-n9sz6" pod "5f84219e-7934-4975-a98f-1d8b04edbc98" (UID: "5f84219e-7934-4975-a98f-1d8b04edbc98"). InnerVolumeSpecName "kube-api-access-n9sz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:30:18.506603 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.506567 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7ltf\" (UniqueName: \"kubernetes.io/projected/32527ca5-6b0f-42c3-a007-da447e507f15-kube-api-access-s7ltf\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:18.506603 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.506598 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32527ca5-6b0f-42c3-a007-da447e507f15-maas-api-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:18.506802 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.506612 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4clbk\" (UniqueName: \"kubernetes.io/projected/dc17776b-6027-4263-b311-5b6c52ce8fc2-kube-api-access-4clbk\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:18.506802 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.506625 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f84219e-7934-4975-a98f-1d8b04edbc98-maas-api-xks-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:18.506802 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.506637 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9sz6\" (UniqueName: \"kubernetes.io/projected/5f84219e-7934-4975-a98f-1d8b04edbc98-kube-api-access-n9sz6\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:18.506802 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.506651 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f84219e-7934-4975-a98f-1d8b04edbc98-maas-api-openshift-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:18.506802 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:18.506663 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc17776b-6027-4263-b311-5b6c52ce8fc2-maas-api-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:19.121148 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.121113 4385 generic.go:356] "Generic (PLEG): container finished" podID="03e6f2ec-efdd-4594-98f0-6ec213f5b29c" containerID="15ac541b4ea227cdc0acf47486265d85c419866cfca23a6b7375d7c7c457d8b5" exitCode=60 Nov 01 06:30:19.121638 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.121187 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v" event={"ID":"03e6f2ec-efdd-4594-98f0-6ec213f5b29c","Type":"ContainerDied","Data":"15ac541b4ea227cdc0acf47486265d85c419866cfca23a6b7375d7c7c457d8b5"} Nov 01 06:30:19.121638 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.121322 4385 scope.go:122] "RemoveContainer" containerID="7360f9f6274d978086c5b4bd7fcbd74f8e4e3640793901edb9b0cc37f6c702bd" Nov 01 06:30:19.122117 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.122093 4385 scope.go:122] "RemoveContainer" containerID="15ac541b4ea227cdc0acf47486265d85c419866cfca23a6b7375d7c7c457d8b5" Nov 01 06:30:19.122970 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:30:19.122944 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891910-xw57v_odh-ai-gateway-infra(03e6f2ec-efdd-4594-98f0-6ec213f5b29c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v" podUID="03e6f2ec-efdd-4594-98f0-6ec213f5b29c" Nov 01 06:30:19.124141 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.124114 4385 generic.go:356] "Generic (PLEG): container finished" podID="6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7" containerID="d3a62d0094ff834c8f7a60af60cfeb2e4fff73a9ab84f545d5be275a840c302f" exitCode=60 Nov 01 06:30:19.124267 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.124221 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" event={"ID":"6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7","Type":"ContainerDied","Data":"d3a62d0094ff834c8f7a60af60cfeb2e4fff73a9ab84f545d5be275a840c302f"} Nov 01 06:30:19.124594 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.124573 4385 scope.go:122] "RemoveContainer" containerID="d3a62d0094ff834c8f7a60af60cfeb2e4fff73a9ab84f545d5be275a840c302f" Nov 01 06:30:19.125398 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:30:19.125375 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc_odh-ai-gateway-infra(6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" podUID="6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7" Nov 01 06:30:19.127421 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.127407 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" Nov 01 06:30:19.127942 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.127450 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786" event={"ID":"32527ca5-6b0f-42c3-a007-da447e507f15","Type":"ContainerDied","Data":"d8bb3a8ac9d84557d82038ff8fa0faca991814d0bbfa10e677bc1fe4dc731a1c"} Nov 01 06:30:19.129686 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.129665 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" Nov 01 06:30:19.129879 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.129664 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr" event={"ID":"dc17776b-6027-4263-b311-5b6c52ce8fc2","Type":"ContainerDied","Data":"e4712199bf777716fcc0551a864be833d9587855b3ccc0be49ee20391821e787"} Nov 01 06:30:19.134205 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.134182 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77" event={"ID":"5f84219e-7934-4975-a98f-1d8b04edbc98","Type":"ContainerDied","Data":"36f0ab2567f8a48410d7cbe825fd15294c95d37e710aa5e02ec5de82b13d9247"} Nov 01 06:30:19.134355 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.134340 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36f0ab2567f8a48410d7cbe825fd15294c95d37e710aa5e02ec5de82b13d9247" Nov 01 06:30:19.134355 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.134296 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77" Nov 01 06:30:19.154358 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.154335 4385 scope.go:122] "RemoveContainer" containerID="fd8dcb2e58d354f50750ba7f7b1c8739b269a21626c81498c795140e15c3ca88" Nov 01 06:30:19.186574 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.186550 4385 scope.go:122] "RemoveContainer" containerID="1386e05228cdd180f5e7d7dd98a758bd1b8949e87f330b67b102ef69bc44ebeb" Nov 01 06:30:19.198894 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.198874 4385 scope.go:122] "RemoveContainer" containerID="898028e5ecc1c190a3d9eddcb1537aa6cfec706cf314ec46fdb9ba734fb10d3a" Nov 01 06:30:19.231087 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.231063 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v"] Nov 01 06:30:19.370604 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.370569 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786"] Nov 01 06:30:19.374720 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.374650 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891910-fx786"] Nov 01 06:30:19.482392 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.482360 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr"] Nov 01 06:30:19.486656 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.486629 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ba2f3-29891910-xnjjr"] Nov 01 06:30:19.556774 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.556743 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:19.569848 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.569806 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:19.628594 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:19.628521 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc"] Nov 01 06:30:20.159566 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:20.159534 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:30:20.326072 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:20.325992 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" Nov 01 06:30:20.350531 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:20.350510 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v" Nov 01 06:30:20.388655 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:20.388617 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32527ca5-6b0f-42c3-a007-da447e507f15" path="/var/lib/kubelet/pods/32527ca5-6b0f-42c3-a007-da447e507f15/volumes" Nov 01 06:30:20.389239 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:20.389220 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc17776b-6027-4263-b311-5b6c52ce8fc2" path="/var/lib/kubelet/pods/dc17776b-6027-4263-b311-5b6c52ce8fc2/volumes" Nov 01 06:30:20.431922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:20.431853 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e6f2ec-efdd-4594-98f0-6ec213f5b29c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e6f2ec-efdd-4594-98f0-6ec213f5b29c-maas-api-service-ca\") pod \"03e6f2ec-efdd-4594-98f0-6ec213f5b29c\" (UID: \"03e6f2ec-efdd-4594-98f0-6ec213f5b29c\") " Nov 01 06:30:20.431922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:20.431902 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7-kube-api-access-zdblx\" (UniqueName: \"kubernetes.io/projected/6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7-kube-api-access-zdblx\") pod \"6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7\" (UID: \"6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7\") " Nov 01 06:30:20.432113 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:20.431947 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7-maas-api-service-ca\") pod \"6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7\" (UID: \"6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7\") " Nov 01 06:30:20.432113 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:20.432003 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e6f2ec-efdd-4594-98f0-6ec213f5b29c-kube-api-access-hkhvf\" (UniqueName: \"kubernetes.io/projected/03e6f2ec-efdd-4594-98f0-6ec213f5b29c-kube-api-access-hkhvf\") pod \"03e6f2ec-efdd-4594-98f0-6ec213f5b29c\" (UID: \"03e6f2ec-efdd-4594-98f0-6ec213f5b29c\") " Nov 01 06:30:20.432224 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:20.432189 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e6f2ec-efdd-4594-98f0-6ec213f5b29c-maas-api-service-ca" pod "03e6f2ec-efdd-4594-98f0-6ec213f5b29c" (UID: "03e6f2ec-efdd-4594-98f0-6ec213f5b29c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:30:20.432375 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:20.432347 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03e6f2ec-efdd-4594-98f0-6ec213f5b29c-maas-api-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:20.432375 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:20.432352 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7-maas-api-service-ca" pod "6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7" (UID: "6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:30:20.434251 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:20.434229 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7-kube-api-access-zdblx" pod "6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7" (UID: "6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7"). InnerVolumeSpecName "kube-api-access-zdblx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:30:20.434359 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:20.434269 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e6f2ec-efdd-4594-98f0-6ec213f5b29c-kube-api-access-hkhvf" pod "03e6f2ec-efdd-4594-98f0-6ec213f5b29c" (UID: "03e6f2ec-efdd-4594-98f0-6ec213f5b29c"). InnerVolumeSpecName "kube-api-access-hkhvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:30:20.533778 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:20.533744 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdblx\" (UniqueName: \"kubernetes.io/projected/6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7-kube-api-access-zdblx\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:20.533778 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:20.533772 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7-maas-api-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:20.533778 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:20.533782 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkhvf\" (UniqueName: \"kubernetes.io/projected/03e6f2ec-efdd-4594-98f0-6ec213f5b29c-kube-api-access-hkhvf\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:30:20.850330 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:20.850294 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:21.011970 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.011886 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:21.168912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.168155 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v" event={"ID":"03e6f2ec-efdd-4594-98f0-6ec213f5b29c","Type":"ContainerDied","Data":"b825b4d05ba7d07f96f493ca981409c90f1a5ee84f1a42536a247dc809d7998f"} Nov 01 06:30:21.168912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.168219 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v" Nov 01 06:30:21.168912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.168231 4385 scope.go:122] "RemoveContainer" containerID="15ac541b4ea227cdc0acf47486265d85c419866cfca23a6b7375d7c7c457d8b5" Nov 01 06:30:21.171395 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.171369 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" Nov 01 06:30:21.171771 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.171748 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc" event={"ID":"6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7","Type":"ContainerDied","Data":"959b90904fff8d1cc9fa10f564122d4614febfccbed17a90f870722e89d4e151"} Nov 01 06:30:21.202073 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.202048 4385 scope.go:122] "RemoveContainer" containerID="d3a62d0094ff834c8f7a60af60cfeb2e4fff73a9ab84f545d5be275a840c302f" Nov 01 06:30:21.240437 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.240397 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc"] Nov 01 06:30:21.241008 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.240988 4385 cpu_manager.go:414] "containerMap: removing container" podUID="6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7" containerName="cleanup" Nov 01 06:30:21.241008 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241009 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7" containerName="cleanup" Nov 01 06:30:21.241181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241024 4385 cpu_manager.go:414] "containerMap: removing container" podUID="03e6f2ec-efdd-4594-98f0-6ec213f5b29c" containerName="cleanup" Nov 01 06:30:21.241181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241031 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e6f2ec-efdd-4594-98f0-6ec213f5b29c" containerName="cleanup" Nov 01 06:30:21.241181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241048 4385 cpu_manager.go:414] "containerMap: removing container" podUID="32527ca5-6b0f-42c3-a007-da447e507f15" containerName="cleanup" Nov 01 06:30:21.241181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241055 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32527ca5-6b0f-42c3-a007-da447e507f15" containerName="cleanup" Nov 01 06:30:21.241181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241064 4385 cpu_manager.go:414] "containerMap: removing container" podUID="3fa42472-eaf4-4a8a-9b47-64a708c379c5" containerName="revoke-keys" Nov 01 06:30:21.241181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241070 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa42472-eaf4-4a8a-9b47-64a708c379c5" containerName="revoke-keys" Nov 01 06:30:21.241181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241089 4385 cpu_manager.go:414] "containerMap: removing container" podUID="32527ca5-6b0f-42c3-a007-da447e507f15" containerName="cleanup" Nov 01 06:30:21.241181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241096 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32527ca5-6b0f-42c3-a007-da447e507f15" containerName="cleanup" Nov 01 06:30:21.241181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241105 4385 cpu_manager.go:414] "containerMap: removing container" podUID="5f84219e-7934-4975-a98f-1d8b04edbc98" containerName="revoke-keys" Nov 01 06:30:21.241181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241112 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f84219e-7934-4975-a98f-1d8b04edbc98" containerName="revoke-keys" Nov 01 06:30:21.241181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241120 4385 cpu_manager.go:414] "containerMap: removing container" podUID="6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7" containerName="cleanup" Nov 01 06:30:21.241181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241127 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7" containerName="cleanup" Nov 01 06:30:21.241181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241140 4385 cpu_manager.go:414] "containerMap: removing container" podUID="03e6f2ec-efdd-4594-98f0-6ec213f5b29c" containerName="cleanup" Nov 01 06:30:21.241181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241148 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e6f2ec-efdd-4594-98f0-6ec213f5b29c" containerName="cleanup" Nov 01 06:30:21.241181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241162 4385 cpu_manager.go:414] "containerMap: removing container" podUID="dc17776b-6027-4263-b311-5b6c52ce8fc2" containerName="cleanup" Nov 01 06:30:21.241181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241168 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc17776b-6027-4263-b311-5b6c52ce8fc2" containerName="cleanup" Nov 01 06:30:21.241181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241183 4385 cpu_manager.go:414] "containerMap: removing container" podUID="dc17776b-6027-4263-b311-5b6c52ce8fc2" containerName="cleanup" Nov 01 06:30:21.241987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241190 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc17776b-6027-4263-b311-5b6c52ce8fc2" containerName="cleanup" Nov 01 06:30:21.241987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241201 4385 cpu_manager.go:414] "containerMap: removing container" podUID="03e6f2ec-efdd-4594-98f0-6ec213f5b29c" containerName="cleanup" Nov 01 06:30:21.241987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241209 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e6f2ec-efdd-4594-98f0-6ec213f5b29c" containerName="cleanup" Nov 01 06:30:21.241987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241219 4385 cpu_manager.go:414] "containerMap: removing container" podUID="dc17776b-6027-4263-b311-5b6c52ce8fc2" containerName="cleanup" Nov 01 06:30:21.241987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241226 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc17776b-6027-4263-b311-5b6c52ce8fc2" containerName="cleanup" Nov 01 06:30:21.241987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241376 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc17776b-6027-4263-b311-5b6c52ce8fc2" containerName="cleanup" Nov 01 06:30:21.241987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241389 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc17776b-6027-4263-b311-5b6c52ce8fc2" containerName="cleanup" Nov 01 06:30:21.241987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241398 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e6f2ec-efdd-4594-98f0-6ec213f5b29c" containerName="cleanup" Nov 01 06:30:21.241987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241409 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc17776b-6027-4263-b311-5b6c52ce8fc2" containerName="cleanup" Nov 01 06:30:21.241987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241419 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7" containerName="cleanup" Nov 01 06:30:21.241987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241430 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="32527ca5-6b0f-42c3-a007-da447e507f15" containerName="cleanup" Nov 01 06:30:21.241987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241440 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7" containerName="cleanup" Nov 01 06:30:21.241987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241450 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f84219e-7934-4975-a98f-1d8b04edbc98" containerName="revoke-keys" Nov 01 06:30:21.241987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241461 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa42472-eaf4-4a8a-9b47-64a708c379c5" containerName="revoke-keys" Nov 01 06:30:21.241987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241471 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="32527ca5-6b0f-42c3-a007-da447e507f15" containerName="cleanup" Nov 01 06:30:21.241987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241479 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7" containerName="cleanup" Nov 01 06:30:21.241987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.241487 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e6f2ec-efdd-4594-98f0-6ec213f5b29c" containerName="cleanup" Nov 01 06:30:21.246979 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.246961 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc" Nov 01 06:30:21.299814 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.299770 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-390a75\"" Nov 01 06:30:21.300012 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.299879 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-390a75-dockercfg-fnc4h\"" Nov 01 06:30:21.304157 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.304035 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc"] Nov 01 06:30:21.344864 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.341599 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc"] Nov 01 06:30:21.344864 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.343036 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dcv6\" (UniqueName: \"kubernetes.io/projected/75f5ca17-d8a4-4ea0-8803-0ad27a3e42db-kube-api-access-8dcv6\") pod \"payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc\" (UID: \"75f5ca17-d8a4-4ea0-8803-0ad27a3e42db\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc" Nov 01 06:30:21.344864 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.343106 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75f5ca17-d8a4-4ea0-8803-0ad27a3e42db-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc\" (UID: \"75f5ca17-d8a4-4ea0-8803-0ad27a3e42db\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc" Nov 01 06:30:21.358980 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.355585 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891910-qtmvc"] Nov 01 06:30:21.418745 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.418692 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk"] Nov 01 06:30:21.425637 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.425550 4385 cpu_manager.go:414] "containerMap: removing container" podUID="32527ca5-6b0f-42c3-a007-da447e507f15" containerName="cleanup" Nov 01 06:30:21.425637 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.425575 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32527ca5-6b0f-42c3-a007-da447e507f15" containerName="cleanup" Nov 01 06:30:21.425889 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.425648 4385 cpu_manager.go:414] "containerMap: removing container" podUID="6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7" containerName="cleanup" Nov 01 06:30:21.425889 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.425657 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7" containerName="cleanup" Nov 01 06:30:21.425889 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.425873 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="32527ca5-6b0f-42c3-a007-da447e507f15" containerName="cleanup" Nov 01 06:30:21.426058 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.425906 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e6f2ec-efdd-4594-98f0-6ec213f5b29c" containerName="cleanup" Nov 01 06:30:21.431282 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.431261 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk" Nov 01 06:30:21.444499 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.444468 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dcv6\" (UniqueName: \"kubernetes.io/projected/75f5ca17-d8a4-4ea0-8803-0ad27a3e42db-kube-api-access-8dcv6\") pod \"payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc\" (UID: \"75f5ca17-d8a4-4ea0-8803-0ad27a3e42db\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc" Nov 01 06:30:21.444619 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.444562 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75f5ca17-d8a4-4ea0-8803-0ad27a3e42db-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc\" (UID: \"75f5ca17-d8a4-4ea0-8803-0ad27a3e42db\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc" Nov 01 06:30:21.445578 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.445561 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75f5ca17-d8a4-4ea0-8803-0ad27a3e42db-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc\" (UID: \"75f5ca17-d8a4-4ea0-8803-0ad27a3e42db\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc" Nov 01 06:30:21.459672 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.459640 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk"] Nov 01 06:30:21.486598 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.480534 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v"] Nov 01 06:30:21.486598 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.486236 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891910-xw57v"] Nov 01 06:30:21.505871 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.504622 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dcv6\" (UniqueName: \"kubernetes.io/projected/75f5ca17-d8a4-4ea0-8803-0ad27a3e42db-kube-api-access-8dcv6\") pod \"payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc\" (UID: \"75f5ca17-d8a4-4ea0-8803-0ad27a3e42db\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc" Nov 01 06:30:21.546515 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.546475 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7gb8\" (UniqueName: \"kubernetes.io/projected/0bf5005a-16d0-42e0-963d-41ca0553856d-kube-api-access-z7gb8\") pod \"payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk\" (UID: \"0bf5005a-16d0-42e0-963d-41ca0553856d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk" Nov 01 06:30:21.546705 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.546588 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf5005a-16d0-42e0-963d-41ca0553856d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk\" (UID: \"0bf5005a-16d0-42e0-963d-41ca0553856d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk" Nov 01 06:30:21.570512 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.570484 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc" Nov 01 06:30:21.651083 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.649109 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7gb8\" (UniqueName: \"kubernetes.io/projected/0bf5005a-16d0-42e0-963d-41ca0553856d-kube-api-access-z7gb8\") pod \"payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk\" (UID: \"0bf5005a-16d0-42e0-963d-41ca0553856d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk" Nov 01 06:30:21.651083 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.649216 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf5005a-16d0-42e0-963d-41ca0553856d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk\" (UID: \"0bf5005a-16d0-42e0-963d-41ca0553856d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk" Nov 01 06:30:21.651083 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.650196 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf5005a-16d0-42e0-963d-41ca0553856d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk\" (UID: \"0bf5005a-16d0-42e0-963d-41ca0553856d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk" Nov 01 06:30:21.707557 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.707495 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7gb8\" (UniqueName: \"kubernetes.io/projected/0bf5005a-16d0-42e0-963d-41ca0553856d-kube-api-access-z7gb8\") pod \"payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk\" (UID: \"0bf5005a-16d0-42e0-963d-41ca0553856d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk" Nov 01 06:30:21.754280 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.754254 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk" Nov 01 06:30:21.963486 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:30:21.963457 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f5ca17_d8a4_4ea0_8803_0ad27a3e42db.slice/crio-17b5b32fcf13156e754178adf818887cbdb0ac1a0a93c60e9569a4c9996ca27b WatchSource:0}: Error finding container 17b5b32fcf13156e754178adf818887cbdb0ac1a0a93c60e9569a4c9996ca27b: Status 404 returned error can't find the container with id 17b5b32fcf13156e754178adf818887cbdb0ac1a0a93c60e9569a4c9996ca27b Nov 01 06:30:21.963755 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:21.963732 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc"] Nov 01 06:30:22.130340 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:30:22.130303 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf5005a_16d0_42e0_963d_41ca0553856d.slice/crio-b8497b9386f83097451a05d6340332a717bac3ba777cc01e9627820091be8450 WatchSource:0}: Error finding container b8497b9386f83097451a05d6340332a717bac3ba777cc01e9627820091be8450: Status 404 returned error can't find the container with id b8497b9386f83097451a05d6340332a717bac3ba777cc01e9627820091be8450 Nov 01 06:30:22.131010 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:22.130988 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk"] Nov 01 06:30:22.184287 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:22.184256 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc" event={"ID":"75f5ca17-d8a4-4ea0-8803-0ad27a3e42db","Type":"ContainerStarted","Data":"1e05e2535af439ac344f9e24f2da2120e7caa560fa227a5f3237c273ad429883"} Nov 01 06:30:22.184701 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:22.184319 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc" event={"ID":"75f5ca17-d8a4-4ea0-8803-0ad27a3e42db","Type":"ContainerStarted","Data":"17b5b32fcf13156e754178adf818887cbdb0ac1a0a93c60e9569a4c9996ca27b"} Nov 01 06:30:22.184701 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:22.184348 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc" Nov 01 06:30:22.186331 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:22.186310 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk" event={"ID":"0bf5005a-16d0-42e0-963d-41ca0553856d","Type":"ContainerStarted","Data":"b8497b9386f83097451a05d6340332a717bac3ba777cc01e9627820091be8450"} Nov 01 06:30:22.298572 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:22.298477 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc" podStartSLOduration=1.298459652 podStartE2EDuration="1.298459652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:22.295051035 +0000 UTC m=+1232.058850893" watchObservedRunningTime="2026-11-01 06:30:22.298459652 +0000 UTC m=+1232.062259509" Nov 01 06:30:22.386754 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:22.386721 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e6f2ec-efdd-4594-98f0-6ec213f5b29c" path="/var/lib/kubelet/pods/03e6f2ec-efdd-4594-98f0-6ec213f5b29c/volumes" Nov 01 06:30:22.387264 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:22.387243 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7" path="/var/lib/kubelet/pods/6b2c76eb-5a4d-4db5-a9f8-63e0e50e1dc7/volumes" Nov 01 06:30:23.027954 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:23.027912 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:23.198631 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:23.198581 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk" event={"ID":"0bf5005a-16d0-42e0-963d-41ca0553856d","Type":"ContainerStarted","Data":"be8b3967e771d44216a80fe2e2ae69a81cbc19f100511c4857eb07701a07a855"} Nov 01 06:30:23.199110 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:23.198898 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk" Nov 01 06:30:23.320911 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:23.315047 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk" podStartSLOduration=2.315023679 podStartE2EDuration="2.315023679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:23.312203162 +0000 UTC m=+1233.076003019" watchObservedRunningTime="2026-11-01 06:30:23.315023679 +0000 UTC m=+1233.078823537" Nov 01 06:30:25.408860 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:25.407922 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" podUID="04ae8d7a-c300-4ddc-89a2-7d3400685bda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 01 06:30:25.545860 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:25.530940 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" podUID="d0029faa-cd51-43d8-8c9f-02399257ea72" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 06:30:25.545860 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:25.541920 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" podUID="4001c2de-286a-4407-9c74-0cc2492bc8ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:30:25.601860 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:25.600903 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" podUID="f35d3f2c-2ea5-47d6-b808-93a12e4923c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:30:25.650862 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:25.619927 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" podUID="6571db97-e767-4917-9543-5ebd4e6ff1e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:30:25.866921 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:25.865917 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" podUID="f5f0b488-c1f8-4a7e-9779-ebf4d94ad037" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:30:28.047976 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:28.047939 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b"] Nov 01 06:30:28.051819 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:28.051793 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b77cc454-l867b"] Nov 01 06:30:28.342324 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:28.342292 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:28.406089 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:28.406047 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="957bb8ef-8ebd-4f16-8c28-254b741c5b5e" path="/var/lib/kubelet/pods/957bb8ef-8ebd-4f16-8c28-254b741c5b5e/volumes" Nov 01 06:30:30.547296 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:30.547261 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:30.739367 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:30.739321 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:33.202978 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:33.202953 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc" Nov 01 06:30:34.210581 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:34.210551 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk" Nov 01 06:30:34.703914 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:34.703824 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:36.866759 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:30:36.866345 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:30:39.687154 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:39.687119 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:40.945961 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:40.945928 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:44.046956 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:44.046923 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87f643dd-4nzbz"] Nov 01 06:30:44.095912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:44.095880 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87f643dd-4nzbz"] Nov 01 06:30:44.387355 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:44.387323 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f986ea82-ba44-4bc9-9eeb-f9f434c6bf19" path="/var/lib/kubelet/pods/f986ea82-ba44-4bc9-9eeb-f9f434c6bf19/volumes" Nov 01 06:30:44.427583 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:44.427551 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:49.230401 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:49.230365 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:49.942214 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:49.942181 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:50.670236 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:50.670201 4385 scope.go:122] "RemoveContainer" containerID="93883262fd44a16701a278f12a65835b61840fb5612fa5109bdd96d4e6633e38" Nov 01 06:30:53.153861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:53.152933 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" podUID="ade9c35c-3890-4fca-844a-7082358cbcf6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:30:53.268967 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:53.268919 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" podUID="695583a3-4ef8-4cad-ba76-69791506f2f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:30:53.413140 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:53.411947 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" podUID="04ae8d7a-c300-4ddc-89a2-7d3400685bda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:30:53.480002 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:53.479961 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" podUID="d0029faa-cd51-43d8-8c9f-02399257ea72" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:30:53.554861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:53.550100 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" podUID="4001c2de-286a-4407-9c74-0cc2492bc8ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:30:53.583037 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:53.582976 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" podUID="f35d3f2c-2ea5-47d6-b808-93a12e4923c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:30:53.867373 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:53.866648 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" podUID="f5f0b488-c1f8-4a7e-9779-ebf4d94ad037" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 06:30:54.453909 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:54.453867 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:54.707078 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:54.706975 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:58.811446 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:58.811402 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:59.050588 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:59.050556 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf59e74e-4c88q"] Nov 01 06:30:59.102261 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:59.102215 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf59e74e-4c88q"] Nov 01 06:30:59.104826 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:59.104783 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:59.254736 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:59.254701 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:59.395799 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:59.395681 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:59.583358 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:59.583284 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:30:59.911219 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:30:59.911177 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:00.390752 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:00.390718 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="381f6bdd-1c00-4f4a-bc28-f901428da687" path="/var/lib/kubelet/pods/381f6bdd-1c00-4f4a-bc28-f901428da687/volumes" Nov 01 06:31:01.414183 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:01.414144 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:02.571448 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.571415 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8"] Nov 01 06:31:02.576267 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.576251 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" Nov 01 06:31:02.635278 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.635246 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8"] Nov 01 06:31:02.677691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.677660 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c79abe\"/\"agc-cred-02725a770fcc\"" Nov 01 06:31:02.679029 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.679002 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c79abe\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c79abe\"" Nov 01 06:31:02.679218 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.679040 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c79abe\"/\"payload-processing-external-model-e2e--a434a17c-dockercfg-wxrtm\"" Nov 01 06:31:02.679304 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.679064 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c79abe\"/\"kube-root-ca.crt\"" Nov 01 06:31:02.679366 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.679335 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c79abe\"/\"routing-overlay\"" Nov 01 06:31:02.679636 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.679619 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c79abe\"/\"openshift-service-ca.crt\"" Nov 01 06:31:02.730534 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.730502 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2k2v\" (UniqueName: \"kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-kube-api-access-k2k2v\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8\" (UID: \"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" Nov 01 06:31:02.730698 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.730559 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8\" (UID: \"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" Nov 01 06:31:02.730749 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.730700 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8\" (UID: \"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" Nov 01 06:31:02.730749 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.730731 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8\" (UID: \"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" Nov 01 06:31:02.831896 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.831873 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8\" (UID: \"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" Nov 01 06:31:02.832057 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.831954 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8\" (UID: \"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" Nov 01 06:31:02.832057 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.831981 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8\" (UID: \"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" Nov 01 06:31:02.832227 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.832139 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2k2v\" (UniqueName: \"kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-kube-api-access-k2k2v\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8\" (UID: \"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" Nov 01 06:31:02.832647 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.832622 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8\" (UID: \"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" Nov 01 06:31:02.832647 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.832640 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8\" (UID: \"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" Nov 01 06:31:02.834612 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.834592 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8\" (UID: \"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" Nov 01 06:31:02.888311 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.888282 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2k2v\" (UniqueName: \"kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-kube-api-access-k2k2v\") pod \"payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8\" (UID: \"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe\") " pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" Nov 01 06:31:02.890158 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:02.890139 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" Nov 01 06:31:03.264206 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:31:03.264166 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38c5e2b_95b1_4080_98f9_8ce28b3ceefe.slice/crio-0925afc873428bd8d0a9a388944cf44694c8d9ee5ab0a0a58e50992ae055b024 WatchSource:0}: Error finding container 0925afc873428bd8d0a9a388944cf44694c8d9ee5ab0a0a58e50992ae055b024: Status 404 returned error can't find the container with id 0925afc873428bd8d0a9a388944cf44694c8d9ee5ab0a0a58e50992ae055b024 Nov 01 06:31:03.264800 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:03.264775 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8"] Nov 01 06:31:03.741036 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:03.741002 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" event={"ID":"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe","Type":"ContainerStarted","Data":"2e48007f786d54cec67cdf478a3dd70a11f0dc9ddd75c9eef1deb685c93aab5d"} Nov 01 06:31:03.741441 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:03.741060 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" Nov 01 06:31:03.741441 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:03.741077 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" event={"ID":"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe","Type":"ContainerStarted","Data":"0925afc873428bd8d0a9a388944cf44694c8d9ee5ab0a0a58e50992ae055b024"} Nov 01 06:31:03.844250 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:03.844205 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd"] Nov 01 06:31:03.848667 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:03.848646 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" Nov 01 06:31:03.856357 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:03.856101 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" podStartSLOduration=1.856089425 podStartE2EDuration="1.856089425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:31:03.853410289 +0000 UTC m=+1273.617210146" watchObservedRunningTime="2026-11-01 06:31:03.856089425 +0000 UTC m=+1273.619889282" Nov 01 06:31:03.900548 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:03.900519 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-e7ab04096de0\"" Nov 01 06:31:03.906277 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:03.906216 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd"] Nov 01 06:31:03.945903 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:03.945852 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f00dce1-27d0-4441-8612-27450fde6103-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd\" (UID: \"8f00dce1-27d0-4441-8612-27450fde6103\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" Nov 01 06:31:03.946067 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:03.946024 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f00dce1-27d0-4441-8612-27450fde6103-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd\" (UID: \"8f00dce1-27d0-4441-8612-27450fde6103\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" Nov 01 06:31:03.946132 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:03.946067 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmg67\" (UniqueName: \"kubernetes.io/projected/8f00dce1-27d0-4441-8612-27450fde6103-kube-api-access-wmg67\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd\" (UID: \"8f00dce1-27d0-4441-8612-27450fde6103\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" Nov 01 06:31:03.946132 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:03.946096 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f00dce1-27d0-4441-8612-27450fde6103-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd\" (UID: \"8f00dce1-27d0-4441-8612-27450fde6103\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" Nov 01 06:31:04.047283 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:04.047190 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f00dce1-27d0-4441-8612-27450fde6103-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd\" (UID: \"8f00dce1-27d0-4441-8612-27450fde6103\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" Nov 01 06:31:04.047629 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:04.047604 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f00dce1-27d0-4441-8612-27450fde6103-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd\" (UID: \"8f00dce1-27d0-4441-8612-27450fde6103\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" Nov 01 06:31:04.049106 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:04.049061 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmg67\" (UniqueName: \"kubernetes.io/projected/8f00dce1-27d0-4441-8612-27450fde6103-kube-api-access-wmg67\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd\" (UID: \"8f00dce1-27d0-4441-8612-27450fde6103\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" Nov 01 06:31:04.049763 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:04.049105 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f00dce1-27d0-4441-8612-27450fde6103-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd\" (UID: \"8f00dce1-27d0-4441-8612-27450fde6103\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" Nov 01 06:31:04.049763 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:04.048201 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f00dce1-27d0-4441-8612-27450fde6103-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd\" (UID: \"8f00dce1-27d0-4441-8612-27450fde6103\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" Nov 01 06:31:04.049763 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:04.048762 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f00dce1-27d0-4441-8612-27450fde6103-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd\" (UID: \"8f00dce1-27d0-4441-8612-27450fde6103\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" Nov 01 06:31:04.052705 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:04.052679 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f00dce1-27d0-4441-8612-27450fde6103-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd\" (UID: \"8f00dce1-27d0-4441-8612-27450fde6103\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" Nov 01 06:31:04.105688 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:04.105651 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmg67\" (UniqueName: \"kubernetes.io/projected/8f00dce1-27d0-4441-8612-27450fde6103-kube-api-access-wmg67\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd\" (UID: \"8f00dce1-27d0-4441-8612-27450fde6103\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" Nov 01 06:31:04.117405 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:04.117376 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:04.163338 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:04.163313 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" Nov 01 06:31:04.324111 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:04.324077 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:04.556018 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:31:04.555988 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f00dce1_27d0_4441_8612_27450fde6103.slice/crio-f0272a88fea9350983787187026fdde4f4e144d22c1220d0d1f45bd8d9f57d6f WatchSource:0}: Error finding container f0272a88fea9350983787187026fdde4f4e144d22c1220d0d1f45bd8d9f57d6f: Status 404 returned error can't find the container with id f0272a88fea9350983787187026fdde4f4e144d22c1220d0d1f45bd8d9f57d6f Nov 01 06:31:04.556342 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:04.556320 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd"] Nov 01 06:31:04.750077 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:04.750042 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" event={"ID":"8f00dce1-27d0-4441-8612-27450fde6103","Type":"ContainerStarted","Data":"bb536fdb0d7e1a16afe4a877d8fc4346be692d6162766766480a0a837adf7093"} Nov 01 06:31:04.750538 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:04.750102 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" Nov 01 06:31:04.750538 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:04.750118 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" event={"ID":"8f00dce1-27d0-4441-8612-27450fde6103","Type":"ContainerStarted","Data":"f0272a88fea9350983787187026fdde4f4e144d22c1220d0d1f45bd8d9f57d6f"} Nov 01 06:31:04.881763 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:04.881627 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" podStartSLOduration=1.8816029140000001 podStartE2EDuration="1.881602914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:31:04.869918256 +0000 UTC m=+1274.633718112" watchObservedRunningTime="2026-11-01 06:31:04.881602914 +0000 UTC m=+1274.645402776" Nov 01 06:31:05.885324 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:05.885288 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:06.055548 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:06.055514 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:06.202912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:06.202037 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:06.835724 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:06.835521 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:31:08.584620 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:08.584574 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:08.728529 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:08.728498 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:10.046564 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:10.046527 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s"] Nov 01 06:31:10.095924 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:10.095887 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5898253b-hgh8s"] Nov 01 06:31:10.384868 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:10.384815 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11580db4-78ae-4c6f-8cfe-68acd757b4c2" path="/var/lib/kubelet/pods/11580db4-78ae-4c6f-8cfe-68acd757b4c2/volumes" Nov 01 06:31:10.934647 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:10.934592 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:11.699433 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:11.699400 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:11.882717 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:11.882682 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:12.243590 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.243560 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v"] Nov 01 06:31:12.248244 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.248215 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v" Nov 01 06:31:12.309616 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.309577 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v"] Nov 01 06:31:12.313200 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.313157 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20dfe3dd-6dab-4021-827a-eed61d2846e7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v\" (UID: \"20dfe3dd-6dab-4021-827a-eed61d2846e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v" Nov 01 06:31:12.313369 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.313275 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc6zl\" (UniqueName: \"kubernetes.io/projected/20dfe3dd-6dab-4021-827a-eed61d2846e7-kube-api-access-dc6zl\") pod \"payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v\" (UID: \"20dfe3dd-6dab-4021-827a-eed61d2846e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v" Nov 01 06:31:12.353289 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.353256 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-25719d\"" Nov 01 06:31:12.353475 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.353303 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-25719d-dockercfg-fmxpk\"" Nov 01 06:31:12.412960 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.412926 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr"] Nov 01 06:31:12.414014 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.413984 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20dfe3dd-6dab-4021-827a-eed61d2846e7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v\" (UID: \"20dfe3dd-6dab-4021-827a-eed61d2846e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v" Nov 01 06:31:12.414099 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.414074 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc6zl\" (UniqueName: \"kubernetes.io/projected/20dfe3dd-6dab-4021-827a-eed61d2846e7-kube-api-access-dc6zl\") pod \"payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v\" (UID: \"20dfe3dd-6dab-4021-827a-eed61d2846e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v" Nov 01 06:31:12.414706 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.414688 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20dfe3dd-6dab-4021-827a-eed61d2846e7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v\" (UID: \"20dfe3dd-6dab-4021-827a-eed61d2846e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v" Nov 01 06:31:12.416879 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.416828 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr" Nov 01 06:31:12.475485 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.475449 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr"] Nov 01 06:31:12.477873 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.477815 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc6zl\" (UniqueName: \"kubernetes.io/projected/20dfe3dd-6dab-4021-827a-eed61d2846e7-kube-api-access-dc6zl\") pod \"payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v\" (UID: \"20dfe3dd-6dab-4021-827a-eed61d2846e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v" Nov 01 06:31:12.515610 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.515516 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d636e4-561d-4a2f-905f-5b864155bb8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr\" (UID: \"43d636e4-561d-4a2f-905f-5b864155bb8c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr" Nov 01 06:31:12.515786 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.515704 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82qxs\" (UniqueName: \"kubernetes.io/projected/43d636e4-561d-4a2f-905f-5b864155bb8c-kube-api-access-82qxs\") pod \"payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr\" (UID: \"43d636e4-561d-4a2f-905f-5b864155bb8c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr" Nov 01 06:31:12.568123 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.568090 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v" Nov 01 06:31:12.617193 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.617151 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82qxs\" (UniqueName: \"kubernetes.io/projected/43d636e4-561d-4a2f-905f-5b864155bb8c-kube-api-access-82qxs\") pod \"payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr\" (UID: \"43d636e4-561d-4a2f-905f-5b864155bb8c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr" Nov 01 06:31:12.617370 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.617306 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d636e4-561d-4a2f-905f-5b864155bb8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr\" (UID: \"43d636e4-561d-4a2f-905f-5b864155bb8c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr" Nov 01 06:31:12.617981 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.617963 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d636e4-561d-4a2f-905f-5b864155bb8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr\" (UID: \"43d636e4-561d-4a2f-905f-5b864155bb8c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr" Nov 01 06:31:12.676887 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.675755 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82qxs\" (UniqueName: \"kubernetes.io/projected/43d636e4-561d-4a2f-905f-5b864155bb8c-kube-api-access-82qxs\") pod \"payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr\" (UID: \"43d636e4-561d-4a2f-905f-5b864155bb8c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr" Nov 01 06:31:12.732419 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.732387 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr" Nov 01 06:31:12.947594 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:31:12.947561 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20dfe3dd_6dab_4021_827a_eed61d2846e7.slice/crio-8429ff03c9d39388896c6c37f4ea81ed116de32de6e7e83fcaaa4a9259f33a41 WatchSource:0}: Error finding container 8429ff03c9d39388896c6c37f4ea81ed116de32de6e7e83fcaaa4a9259f33a41: Status 404 returned error can't find the container with id 8429ff03c9d39388896c6c37f4ea81ed116de32de6e7e83fcaaa4a9259f33a41 Nov 01 06:31:12.948193 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:12.948170 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v"] Nov 01 06:31:13.111467 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:31:13.111438 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d636e4_561d_4a2f_905f_5b864155bb8c.slice/crio-463f9ef304455dd640e70c9e5930116ef619df1c3ad51120397f380978f1f43e WatchSource:0}: Error finding container 463f9ef304455dd640e70c9e5930116ef619df1c3ad51120397f380978f1f43e: Status 404 returned error can't find the container with id 463f9ef304455dd640e70c9e5930116ef619df1c3ad51120397f380978f1f43e Nov 01 06:31:13.113772 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:13.113741 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr"] Nov 01 06:31:13.562536 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:13.562426 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:13.701442 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:13.701381 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:13.868071 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:13.868036 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v" event={"ID":"20dfe3dd-6dab-4021-827a-eed61d2846e7","Type":"ContainerStarted","Data":"3ce7b00e13433f3aa9a72745f7764156871b7716e41333533f16854dcc6a7feb"} Nov 01 06:31:13.868553 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:13.868088 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v" event={"ID":"20dfe3dd-6dab-4021-827a-eed61d2846e7","Type":"ContainerStarted","Data":"8429ff03c9d39388896c6c37f4ea81ed116de32de6e7e83fcaaa4a9259f33a41"} Nov 01 06:31:13.868553 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:13.868133 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v" Nov 01 06:31:13.870114 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:13.870093 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr" event={"ID":"43d636e4-561d-4a2f-905f-5b864155bb8c","Type":"ContainerStarted","Data":"bf00a0e4935f5301839909779b8f845d66e7a3258f75d62484d54e4090596285"} Nov 01 06:31:13.870247 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:13.870127 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr" event={"ID":"43d636e4-561d-4a2f-905f-5b864155bb8c","Type":"ContainerStarted","Data":"463f9ef304455dd640e70c9e5930116ef619df1c3ad51120397f380978f1f43e"} Nov 01 06:31:13.870247 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:13.870186 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr" Nov 01 06:31:13.982238 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:13.982185 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v" podStartSLOduration=1.982167353 podStartE2EDuration="1.982167353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:31:13.979563167 +0000 UTC m=+1283.743363024" watchObservedRunningTime="2026-11-01 06:31:13.982167353 +0000 UTC m=+1283.745967211" Nov 01 06:31:14.053269 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:14.053235 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd"] Nov 01 06:31:14.056058 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:14.056035 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" Nov 01 06:31:14.094552 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:14.094505 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr" podStartSLOduration=2.094491842 podStartE2EDuration="2.094491842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:31:14.093748499 +0000 UTC m=+1283.857548357" watchObservedRunningTime="2026-11-01 06:31:14.094491842 +0000 UTC m=+1283.858291745" Nov 01 06:31:14.396621 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:14.396592 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d"] Nov 01 06:31:14.396810 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:14.396763 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" podUID="f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" containerName="istio-proxy" containerID="cri-o://e3c4df095ed9da14695b370c3bea7d3ea226872d3744d6b2f570de1bc2ebda07" gracePeriod=30 Nov 01 06:31:14.753488 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:14.753402 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" Nov 01 06:31:14.851184 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:14.851142 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c79abe/agc-cred-02725a770fcc: secret "agc-cred-02725a770fcc" not found Nov 01 06:31:14.851184 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:14.851177 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8: secret "agc-cred-02725a770fcc" not found Nov 01 06:31:14.851446 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:14.851260 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-provider-credentials podName:a38c5e2b-95b1-4080-98f9-8ce28b3ceefe nodeName:}" failed. No retries permitted until 2026-11-01 06:31:15.351239107 +0000 UTC m=+1285.115038959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" (UID: "a38c5e2b-95b1-4080-98f9-8ce28b3ceefe") : secret "agc-cred-02725a770fcc" not found Nov 01 06:31:15.357557 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:15.357529 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c79abe/agc-cred-02725a770fcc: secret "agc-cred-02725a770fcc" not found Nov 01 06:31:15.357557 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:15.357554 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8: secret "agc-cred-02725a770fcc" not found Nov 01 06:31:15.357994 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:15.357608 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-provider-credentials podName:a38c5e2b-95b1-4080-98f9-8ce28b3ceefe nodeName:}" failed. No retries permitted until 2026-11-01 06:31:16.357594207 +0000 UTC m=+1286.121394042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" (UID: "a38c5e2b-95b1-4080-98f9-8ce28b3ceefe") : secret "agc-cred-02725a770fcc" not found Nov 01 06:31:15.492563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:15.492523 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8"] Nov 01 06:31:15.652741 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:15.652652 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:16.369165 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:16.369133 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c79abe/agc-cred-02725a770fcc: secret "agc-cred-02725a770fcc" not found Nov 01 06:31:16.369165 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:16.369167 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8: secret "agc-cred-02725a770fcc" not found Nov 01 06:31:16.369768 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:16.369239 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-provider-credentials podName:a38c5e2b-95b1-4080-98f9-8ce28b3ceefe nodeName:}" failed. No retries permitted until 2026-11-01 06:31:18.369217717 +0000 UTC m=+1288.133017558 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" (UID: "a38c5e2b-95b1-4080-98f9-8ce28b3ceefe") : secret "agc-cred-02725a770fcc" not found Nov 01 06:31:16.747545 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:16.747457 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:18.394971 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:18.394941 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c79abe/agc-cred-02725a770fcc: secret "agc-cred-02725a770fcc" not found Nov 01 06:31:18.394971 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:18.394968 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8: secret "agc-cred-02725a770fcc" not found Nov 01 06:31:18.395532 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:18.395032 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-provider-credentials podName:a38c5e2b-95b1-4080-98f9-8ce28b3ceefe nodeName:}" failed. No retries permitted until 2026-11-01 06:31:22.395013647 +0000 UTC m=+1292.158813495 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" (UID: "a38c5e2b-95b1-4080-98f9-8ce28b3ceefe") : secret "agc-cred-02725a770fcc" not found Nov 01 06:31:19.053966 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.053922 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" podUID="8f00dce1-27d0-4441-8612-27450fde6103" containerName="payload-processing" containerID="cri-o://bb536fdb0d7e1a16afe4a877d8fc4346be692d6162766766480a0a837adf7093" gracePeriod=25 Nov 01 06:31:19.119827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.119792 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59"] Nov 01 06:31:19.120054 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.120026 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" podUID="7b22ecfe-990b-49a7-b565-f8b94668a5a1" containerName="istio-proxy" containerID="cri-o://76e8fb39e36691f5e94b922eec96b1ee3657a0222edee27d8d0cbea5a980d460" gracePeriod=30 Nov 01 06:31:19.335658 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.335635 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" Nov 01 06:31:19.408875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.408069 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f00dce1-27d0-4441-8612-27450fde6103-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f00dce1-27d0-4441-8612-27450fde6103-routing-overlay-volume\") pod \"8f00dce1-27d0-4441-8612-27450fde6103\" (UID: \"8f00dce1-27d0-4441-8612-27450fde6103\") " Nov 01 06:31:19.408875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.408165 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f00dce1-27d0-4441-8612-27450fde6103-kube-api-access-wmg67\" (UniqueName: \"kubernetes.io/projected/8f00dce1-27d0-4441-8612-27450fde6103-kube-api-access-wmg67\") pod \"8f00dce1-27d0-4441-8612-27450fde6103\" (UID: \"8f00dce1-27d0-4441-8612-27450fde6103\") " Nov 01 06:31:19.408875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.408240 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f00dce1-27d0-4441-8612-27450fde6103-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f00dce1-27d0-4441-8612-27450fde6103-provider-credentials\") pod \"8f00dce1-27d0-4441-8612-27450fde6103\" (UID: \"8f00dce1-27d0-4441-8612-27450fde6103\") " Nov 01 06:31:19.408875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.408361 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f00dce1-27d0-4441-8612-27450fde6103-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f00dce1-27d0-4441-8612-27450fde6103-plugins-config-volume\") pod \"8f00dce1-27d0-4441-8612-27450fde6103\" (UID: \"8f00dce1-27d0-4441-8612-27450fde6103\") " Nov 01 06:31:19.408875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.408429 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f00dce1-27d0-4441-8612-27450fde6103-routing-overlay-volume" pod "8f00dce1-27d0-4441-8612-27450fde6103" (UID: "8f00dce1-27d0-4441-8612-27450fde6103"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:31:19.409395 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.408914 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f00dce1-27d0-4441-8612-27450fde6103-plugins-config-volume" pod "8f00dce1-27d0-4441-8612-27450fde6103" (UID: "8f00dce1-27d0-4441-8612-27450fde6103"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:31:19.409395 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.409022 4385 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f00dce1-27d0-4441-8612-27450fde6103-routing-overlay-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:19.411132 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.411106 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f00dce1-27d0-4441-8612-27450fde6103-kube-api-access-wmg67" pod "8f00dce1-27d0-4441-8612-27450fde6103" (UID: "8f00dce1-27d0-4441-8612-27450fde6103"). InnerVolumeSpecName "kube-api-access-wmg67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:31:19.411371 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.411349 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f00dce1-27d0-4441-8612-27450fde6103-provider-credentials" pod "8f00dce1-27d0-4441-8612-27450fde6103" (UID: "8f00dce1-27d0-4441-8612-27450fde6103"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:31:19.509930 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.509903 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmg67\" (UniqueName: \"kubernetes.io/projected/8f00dce1-27d0-4441-8612-27450fde6103-kube-api-access-wmg67\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:19.510006 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.509932 4385 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f00dce1-27d0-4441-8612-27450fde6103-provider-credentials\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:19.510006 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.509943 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f00dce1-27d0-4441-8612-27450fde6103-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:19.653261 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.653236 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:31:19.712387 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.712358 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-workload-socket\") pod \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " Nov 01 06:31:19.712387 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.712400 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istiod-ca-cert\") pod \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " Nov 01 06:31:19.712643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.712459 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-data\") pod \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " Nov 01 06:31:19.712643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.712505 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-envoy\") pod \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " Nov 01 06:31:19.712643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.712554 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-workload-certs\") pod \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " Nov 01 06:31:19.712804 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.712641 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-workload-socket" pod "f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" (UID: "f7f4ac4b-38cb-4b95-9ec4-88b7d3291126"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:31:19.712804 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.712678 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-podinfo\") pod \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " Nov 01 06:31:19.712804 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.712749 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-token\" (UniqueName: \"kubernetes.io/projected/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-token\") pod \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " Nov 01 06:31:19.712804 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.712798 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-kube-api-access-dg7jm\" (UniqueName: \"kubernetes.io/projected/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-kube-api-access-dg7jm\") pod \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " Nov 01 06:31:19.713034 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.712810 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-data" pod "f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" (UID: "f7f4ac4b-38cb-4b95-9ec4-88b7d3291126"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:31:19.713034 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.712898 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-workload-certs" pod "f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" (UID: "f7f4ac4b-38cb-4b95-9ec4-88b7d3291126"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:31:19.713034 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.712928 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istiod-ca-cert" pod "f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" (UID: "f7f4ac4b-38cb-4b95-9ec4-88b7d3291126"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:31:19.713034 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.712980 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-credential-socket\") pod \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\" (UID: \"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126\") " Nov 01 06:31:19.713381 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.713339 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-credential-socket" pod "f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" (UID: "f7f4ac4b-38cb-4b95-9ec4-88b7d3291126"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:31:19.713728 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.713629 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:19.713728 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.713656 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:19.713728 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.713671 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:19.713728 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.713685 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:19.713728 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.713699 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:19.715261 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.715236 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-kube-api-access-dg7jm" pod "f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" (UID: "f7f4ac4b-38cb-4b95-9ec4-88b7d3291126"). InnerVolumeSpecName "kube-api-access-dg7jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:31:19.715479 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.715453 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-podinfo" pod "f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" (UID: "f7f4ac4b-38cb-4b95-9ec4-88b7d3291126"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:31:19.715669 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.715648 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-token" pod "f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" (UID: "f7f4ac4b-38cb-4b95-9ec4-88b7d3291126"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:31:19.715763 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.715701 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-envoy" pod "f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" (UID: "f7f4ac4b-38cb-4b95-9ec4-88b7d3291126"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:31:19.815150 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.815113 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg7jm\" (UniqueName: \"kubernetes.io/projected/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-kube-api-access-dg7jm\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:19.815150 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.815149 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:19.815150 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.815159 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:19.815459 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.815168 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:19.921556 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.921522 4385 generic.go:356] "Generic (PLEG): container finished" podID="f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" containerID="e3c4df095ed9da14695b370c3bea7d3ea226872d3744d6b2f570de1bc2ebda07" exitCode=0 Nov 01 06:31:19.921556 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.921569 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" Nov 01 06:31:19.921828 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.921604 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" event={"ID":"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126","Type":"ContainerDied","Data":"e3c4df095ed9da14695b370c3bea7d3ea226872d3744d6b2f570de1bc2ebda07"} Nov 01 06:31:19.921828 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.921696 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d" event={"ID":"f7f4ac4b-38cb-4b95-9ec4-88b7d3291126","Type":"ContainerDied","Data":"89a5da19e916c70209f550bb7b878112d1847a8b031d74c55126f3c55fade2cb"} Nov 01 06:31:19.921828 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.921732 4385 scope.go:122] "RemoveContainer" containerID="e3c4df095ed9da14695b370c3bea7d3ea226872d3744d6b2f570de1bc2ebda07" Nov 01 06:31:19.923191 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.923165 4385 generic.go:356] "Generic (PLEG): container finished" podID="8f00dce1-27d0-4441-8612-27450fde6103" containerID="bb536fdb0d7e1a16afe4a877d8fc4346be692d6162766766480a0a837adf7093" exitCode=0 Nov 01 06:31:19.923340 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.923196 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" event={"ID":"8f00dce1-27d0-4441-8612-27450fde6103","Type":"ContainerDied","Data":"bb536fdb0d7e1a16afe4a877d8fc4346be692d6162766766480a0a837adf7093"} Nov 01 06:31:19.923340 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.923235 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" Nov 01 06:31:19.923340 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.923246 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd" event={"ID":"8f00dce1-27d0-4441-8612-27450fde6103","Type":"ContainerDied","Data":"f0272a88fea9350983787187026fdde4f4e144d22c1220d0d1f45bd8d9f57d6f"} Nov 01 06:31:19.933906 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.933886 4385 scope.go:122] "RemoveContainer" containerID="e3c4df095ed9da14695b370c3bea7d3ea226872d3744d6b2f570de1bc2ebda07" Nov 01 06:31:19.934188 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:19.934167 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c4df095ed9da14695b370c3bea7d3ea226872d3744d6b2f570de1bc2ebda07\": container with ID starting with e3c4df095ed9da14695b370c3bea7d3ea226872d3744d6b2f570de1bc2ebda07 not found: ID does not exist" containerID="e3c4df095ed9da14695b370c3bea7d3ea226872d3744d6b2f570de1bc2ebda07" Nov 01 06:31:19.934260 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.934197 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c4df095ed9da14695b370c3bea7d3ea226872d3744d6b2f570de1bc2ebda07"} err="failed to get container status \"e3c4df095ed9da14695b370c3bea7d3ea226872d3744d6b2f570de1bc2ebda07\": rpc error: code = NotFound desc = could not find container \"e3c4df095ed9da14695b370c3bea7d3ea226872d3744d6b2f570de1bc2ebda07\": container with ID starting with e3c4df095ed9da14695b370c3bea7d3ea226872d3744d6b2f570de1bc2ebda07 not found: ID does not exist" Nov 01 06:31:19.934260 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.934214 4385 scope.go:122] "RemoveContainer" containerID="bb536fdb0d7e1a16afe4a877d8fc4346be692d6162766766480a0a837adf7093" Nov 01 06:31:19.946402 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.946385 4385 scope.go:122] "RemoveContainer" containerID="bb536fdb0d7e1a16afe4a877d8fc4346be692d6162766766480a0a837adf7093" Nov 01 06:31:19.946915 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:19.946888 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb536fdb0d7e1a16afe4a877d8fc4346be692d6162766766480a0a837adf7093\": container with ID starting with bb536fdb0d7e1a16afe4a877d8fc4346be692d6162766766480a0a837adf7093 not found: ID does not exist" containerID="bb536fdb0d7e1a16afe4a877d8fc4346be692d6162766766480a0a837adf7093" Nov 01 06:31:19.947000 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:19.946926 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb536fdb0d7e1a16afe4a877d8fc4346be692d6162766766480a0a837adf7093"} err="failed to get container status \"bb536fdb0d7e1a16afe4a877d8fc4346be692d6162766766480a0a837adf7093\": rpc error: code = NotFound desc = could not find container \"bb536fdb0d7e1a16afe4a877d8fc4346be692d6162766766480a0a837adf7093\": container with ID starting with bb536fdb0d7e1a16afe4a877d8fc4346be692d6162766766480a0a837adf7093 not found: ID does not exist" Nov 01 06:31:20.087957 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.087930 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd"] Nov 01 06:31:20.091663 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.091636 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-d87bqn8hd"] Nov 01 06:31:20.197358 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.197267 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d"] Nov 01 06:31:20.203533 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.203502 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-26j9d"] Nov 01 06:31:20.389449 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.389412 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f00dce1-27d0-4441-8612-27450fde6103" path="/var/lib/kubelet/pods/8f00dce1-27d0-4441-8612-27450fde6103/volumes" Nov 01 06:31:20.390160 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.390138 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" path="/var/lib/kubelet/pods/f7f4ac4b-38cb-4b95-9ec4-88b7d3291126/volumes" Nov 01 06:31:20.492894 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.492776 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" podUID="a38c5e2b-95b1-4080-98f9-8ce28b3ceefe" containerName="payload-processing" containerID="cri-o://2e48007f786d54cec67cdf478a3dd70a11f0dc9ddd75c9eef1deb685c93aab5d" gracePeriod=25 Nov 01 06:31:20.770654 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.770632 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" Nov 01 06:31:20.935346 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.935315 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-routing-overlay-volume\") pod \"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe\" (UID: \"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe\") " Nov 01 06:31:20.935518 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.935429 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-plugins-config-volume\") pod \"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe\" (UID: \"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe\") " Nov 01 06:31:20.935518 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.935489 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-kube-api-access-k2k2v\" (UniqueName: \"kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-kube-api-access-k2k2v\") pod \"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe\" (UID: \"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe\") " Nov 01 06:31:20.935643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.935606 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-provider-credentials\") pod \"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe\" (UID: \"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe\") " Nov 01 06:31:20.936214 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.936177 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-plugins-config-volume" pod "a38c5e2b-95b1-4080-98f9-8ce28b3ceefe" (UID: "a38c5e2b-95b1-4080-98f9-8ce28b3ceefe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:31:20.944898 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.936547 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-routing-overlay-volume" pod "a38c5e2b-95b1-4080-98f9-8ce28b3ceefe" (UID: "a38c5e2b-95b1-4080-98f9-8ce28b3ceefe"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:31:20.947307 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.947076 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-kube-api-access-k2k2v" pod "a38c5e2b-95b1-4080-98f9-8ce28b3ceefe" (UID: "a38c5e2b-95b1-4080-98f9-8ce28b3ceefe"). InnerVolumeSpecName "kube-api-access-k2k2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:31:20.947597 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.947575 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-provider-credentials" pod "a38c5e2b-95b1-4080-98f9-8ce28b3ceefe" (UID: "a38c5e2b-95b1-4080-98f9-8ce28b3ceefe"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:31:20.948727 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.948671 4385 generic.go:356] "Generic (PLEG): container finished" podID="a38c5e2b-95b1-4080-98f9-8ce28b3ceefe" containerID="2e48007f786d54cec67cdf478a3dd70a11f0dc9ddd75c9eef1deb685c93aab5d" exitCode=0 Nov 01 06:31:20.948969 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.948824 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" event={"ID":"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe","Type":"ContainerDied","Data":"2e48007f786d54cec67cdf478a3dd70a11f0dc9ddd75c9eef1deb685c93aab5d"} Nov 01 06:31:20.948969 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.948916 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" event={"ID":"a38c5e2b-95b1-4080-98f9-8ce28b3ceefe","Type":"ContainerDied","Data":"0925afc873428bd8d0a9a388944cf44694c8d9ee5ab0a0a58e50992ae055b024"} Nov 01 06:31:20.948969 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.948948 4385 scope.go:122] "RemoveContainer" containerID="2e48007f786d54cec67cdf478a3dd70a11f0dc9ddd75c9eef1deb685c93aab5d" Nov 01 06:31:20.949159 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.949065 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8" Nov 01 06:31:20.966699 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.966681 4385 scope.go:122] "RemoveContainer" containerID="2e48007f786d54cec67cdf478a3dd70a11f0dc9ddd75c9eef1deb685c93aab5d" Nov 01 06:31:20.967001 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:20.966972 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e48007f786d54cec67cdf478a3dd70a11f0dc9ddd75c9eef1deb685c93aab5d\": container with ID starting with 2e48007f786d54cec67cdf478a3dd70a11f0dc9ddd75c9eef1deb685c93aab5d not found: ID does not exist" containerID="2e48007f786d54cec67cdf478a3dd70a11f0dc9ddd75c9eef1deb685c93aab5d" Nov 01 06:31:20.967091 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:20.967018 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e48007f786d54cec67cdf478a3dd70a11f0dc9ddd75c9eef1deb685c93aab5d"} err="failed to get container status \"2e48007f786d54cec67cdf478a3dd70a11f0dc9ddd75c9eef1deb685c93aab5d\": rpc error: code = NotFound desc = could not find container \"2e48007f786d54cec67cdf478a3dd70a11f0dc9ddd75c9eef1deb685c93aab5d\": container with ID starting with 2e48007f786d54cec67cdf478a3dd70a11f0dc9ddd75c9eef1deb685c93aab5d not found: ID does not exist" Nov 01 06:31:21.037297 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:21.037219 4385 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-provider-credentials\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:21.037297 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:21.037250 4385 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-routing-overlay-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:21.037297 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:21.037268 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:21.037297 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:21.037281 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2k2v\" (UniqueName: \"kubernetes.io/projected/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe-kube-api-access-k2k2v\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:21.149918 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:21.147538 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8"] Nov 01 06:31:21.153071 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:21.153037 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c79abe/payload-processing-external-model-e2e-extmod-a-c79abe-7dc9p4pv8"] Nov 01 06:31:21.767572 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:21.767528 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:22.413649 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:22.413606 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a38c5e2b-95b1-4080-98f9-8ce28b3ceefe" path="/var/lib/kubelet/pods/a38c5e2b-95b1-4080-98f9-8ce28b3ceefe/volumes" Nov 01 06:31:23.417353 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:23.417318 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:23.574859 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:23.574788 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:23.828817 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:23.828783 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:24.396358 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.396285 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:31:24.485628 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.485590 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-data\") pod \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " Nov 01 06:31:24.486157 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.485671 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-workload-certs\") pod \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " Nov 01 06:31:24.486157 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.485711 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-token\" (UniqueName: \"kubernetes.io/projected/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-token\") pod \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " Nov 01 06:31:24.486157 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.485757 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-credential-socket\") pod \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " Nov 01 06:31:24.486157 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.485786 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b22ecfe-990b-49a7-b565-f8b94668a5a1-kube-api-access-kvvxx\" (UniqueName: \"kubernetes.io/projected/7b22ecfe-990b-49a7-b565-f8b94668a5a1-kube-api-access-kvvxx\") pod \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " Nov 01 06:31:24.486157 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.485919 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-data" pod "7b22ecfe-990b-49a7-b565-f8b94668a5a1" (UID: "7b22ecfe-990b-49a7-b565-f8b94668a5a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:31:24.486157 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.485966 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-envoy\") pod \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " Nov 01 06:31:24.486157 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.485978 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-workload-certs" pod "7b22ecfe-990b-49a7-b565-f8b94668a5a1" (UID: "7b22ecfe-990b-49a7-b565-f8b94668a5a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:31:24.486157 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.485996 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istiod-ca-cert\") pod \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " Nov 01 06:31:24.486157 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.486061 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-podinfo\") pod \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " Nov 01 06:31:24.486157 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.486139 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-credential-socket" pod "7b22ecfe-990b-49a7-b565-f8b94668a5a1" (UID: "7b22ecfe-990b-49a7-b565-f8b94668a5a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:31:24.486811 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.486169 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-workload-socket\") pod \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\" (UID: \"7b22ecfe-990b-49a7-b565-f8b94668a5a1\") " Nov 01 06:31:24.486811 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.486512 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istiod-ca-cert" pod "7b22ecfe-990b-49a7-b565-f8b94668a5a1" (UID: "7b22ecfe-990b-49a7-b565-f8b94668a5a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:31:24.487063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.487041 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:24.487124 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.487069 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:24.487124 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.487085 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:24.487124 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.487099 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:24.487460 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.487433 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-workload-socket" pod "7b22ecfe-990b-49a7-b565-f8b94668a5a1" (UID: "7b22ecfe-990b-49a7-b565-f8b94668a5a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:31:24.488687 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.488661 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-envoy" pod "7b22ecfe-990b-49a7-b565-f8b94668a5a1" (UID: "7b22ecfe-990b-49a7-b565-f8b94668a5a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:31:24.488986 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.488959 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b22ecfe-990b-49a7-b565-f8b94668a5a1-kube-api-access-kvvxx" pod "7b22ecfe-990b-49a7-b565-f8b94668a5a1" (UID: "7b22ecfe-990b-49a7-b565-f8b94668a5a1"). InnerVolumeSpecName "kube-api-access-kvvxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:31:24.489593 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.489568 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-token" pod "7b22ecfe-990b-49a7-b565-f8b94668a5a1" (UID: "7b22ecfe-990b-49a7-b565-f8b94668a5a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:31:24.489745 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.489722 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-podinfo" pod "7b22ecfe-990b-49a7-b565-f8b94668a5a1" (UID: "7b22ecfe-990b-49a7-b565-f8b94668a5a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:31:24.588918 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.588875 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:24.588918 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.588910 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvvxx\" (UniqueName: \"kubernetes.io/projected/7b22ecfe-990b-49a7-b565-f8b94668a5a1-kube-api-access-kvvxx\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:24.588918 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.588924 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:24.589171 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.588937 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b22ecfe-990b-49a7-b565-f8b94668a5a1-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:24.589171 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.588949 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b22ecfe-990b-49a7-b565-f8b94668a5a1-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:24.884617 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.884584 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v" Nov 01 06:31:24.884821 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.884664 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr" Nov 01 06:31:24.894399 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:24.894369 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:25.009515 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:25.009464 4385 generic.go:356] "Generic (PLEG): container finished" podID="7b22ecfe-990b-49a7-b565-f8b94668a5a1" containerID="76e8fb39e36691f5e94b922eec96b1ee3657a0222edee27d8d0cbea5a980d460" exitCode=0 Nov 01 06:31:25.009712 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:25.009532 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" event={"ID":"7b22ecfe-990b-49a7-b565-f8b94668a5a1","Type":"ContainerDied","Data":"76e8fb39e36691f5e94b922eec96b1ee3657a0222edee27d8d0cbea5a980d460"} Nov 01 06:31:25.009712 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:25.009583 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" event={"ID":"7b22ecfe-990b-49a7-b565-f8b94668a5a1","Type":"ContainerDied","Data":"17452e991af4c04557c4dd566373319a8c01b65e7b2973ed0f0b11255f933941"} Nov 01 06:31:25.009712 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:25.009609 4385 scope.go:122] "RemoveContainer" containerID="76e8fb39e36691f5e94b922eec96b1ee3657a0222edee27d8d0cbea5a980d460" Nov 01 06:31:25.009926 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:25.009756 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59" Nov 01 06:31:25.031149 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:25.031125 4385 scope.go:122] "RemoveContainer" containerID="76e8fb39e36691f5e94b922eec96b1ee3657a0222edee27d8d0cbea5a980d460" Nov 01 06:31:25.033902 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:25.031670 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e8fb39e36691f5e94b922eec96b1ee3657a0222edee27d8d0cbea5a980d460\": container with ID starting with 76e8fb39e36691f5e94b922eec96b1ee3657a0222edee27d8d0cbea5a980d460 not found: ID does not exist" containerID="76e8fb39e36691f5e94b922eec96b1ee3657a0222edee27d8d0cbea5a980d460" Nov 01 06:31:25.033902 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:25.031711 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e8fb39e36691f5e94b922eec96b1ee3657a0222edee27d8d0cbea5a980d460"} err="failed to get container status \"76e8fb39e36691f5e94b922eec96b1ee3657a0222edee27d8d0cbea5a980d460\": rpc error: code = NotFound desc = could not find container \"76e8fb39e36691f5e94b922eec96b1ee3657a0222edee27d8d0cbea5a980d460\": container with ID starting with 76e8fb39e36691f5e94b922eec96b1ee3657a0222edee27d8d0cbea5a980d460 not found: ID does not exist" Nov 01 06:31:25.048702 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:25.048673 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:25.230199 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:25.226887 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59"] Nov 01 06:31:25.232376 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:25.232349 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kvv59"] Nov 01 06:31:26.045939 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:26.045908 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h"] Nov 01 06:31:26.049971 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:26.049944 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81031788-qdj4h"] Nov 01 06:31:26.395609 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:26.395571 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84" path="/var/lib/kubelet/pods/0cf42b85-afb1-4d18-ae1a-9d73f1cbaf84/volumes" Nov 01 06:31:26.396223 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:26.396206 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b22ecfe-990b-49a7-b565-f8b94668a5a1" path="/var/lib/kubelet/pods/7b22ecfe-990b-49a7-b565-f8b94668a5a1/volumes" Nov 01 06:31:26.763541 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:26.763457 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:27.563271 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:27.563240 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:27.754071 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:27.754036 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:28.356722 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:28.356691 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:29.500181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:29.500145 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:31.516450 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.516367 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg"] Nov 01 06:31:31.517091 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.517072 4385 cpu_manager.go:414] "containerMap: removing container" podUID="f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" containerName="istio-proxy" Nov 01 06:31:31.517181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.517092 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" containerName="istio-proxy" Nov 01 06:31:31.517181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.517118 4385 cpu_manager.go:414] "containerMap: removing container" podUID="7b22ecfe-990b-49a7-b565-f8b94668a5a1" containerName="istio-proxy" Nov 01 06:31:31.517181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.517126 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b22ecfe-990b-49a7-b565-f8b94668a5a1" containerName="istio-proxy" Nov 01 06:31:31.517181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.517141 4385 cpu_manager.go:414] "containerMap: removing container" podUID="8f00dce1-27d0-4441-8612-27450fde6103" containerName="payload-processing" Nov 01 06:31:31.517181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.517150 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f00dce1-27d0-4441-8612-27450fde6103" containerName="payload-processing" Nov 01 06:31:31.517495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.517191 4385 cpu_manager.go:414] "containerMap: removing container" podUID="a38c5e2b-95b1-4080-98f9-8ce28b3ceefe" containerName="payload-processing" Nov 01 06:31:31.517495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.517198 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a38c5e2b-95b1-4080-98f9-8ce28b3ceefe" containerName="payload-processing" Nov 01 06:31:31.517495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.517337 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="a38c5e2b-95b1-4080-98f9-8ce28b3ceefe" containerName="payload-processing" Nov 01 06:31:31.517495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.517351 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f4ac4b-38cb-4b95-9ec4-88b7d3291126" containerName="istio-proxy" Nov 01 06:31:31.517495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.517365 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b22ecfe-990b-49a7-b565-f8b94668a5a1" containerName="istio-proxy" Nov 01 06:31:31.517495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.517374 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f00dce1-27d0-4441-8612-27450fde6103" containerName="payload-processing" Nov 01 06:31:31.522386 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.522365 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" Nov 01 06:31:31.574067 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.574035 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-c8aede0ff036\"" Nov 01 06:31:31.577475 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.577451 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg"] Nov 01 06:31:31.670081 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.670025 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg\" (UID: \"d8da888f-3fd6-4d71-a5be-ad96ea3fd427\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" Nov 01 06:31:31.670302 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.670101 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg\" (UID: \"d8da888f-3fd6-4d71-a5be-ad96ea3fd427\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" Nov 01 06:31:31.670302 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.670156 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg\" (UID: \"d8da888f-3fd6-4d71-a5be-ad96ea3fd427\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" Nov 01 06:31:31.670302 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.670266 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw6f5\" (UniqueName: \"kubernetes.io/projected/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-kube-api-access-dw6f5\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg\" (UID: \"d8da888f-3fd6-4d71-a5be-ad96ea3fd427\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" Nov 01 06:31:31.771706 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.771621 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg\" (UID: \"d8da888f-3fd6-4d71-a5be-ad96ea3fd427\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" Nov 01 06:31:31.771706 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.771660 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg\" (UID: \"d8da888f-3fd6-4d71-a5be-ad96ea3fd427\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" Nov 01 06:31:31.771949 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.771716 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw6f5\" (UniqueName: \"kubernetes.io/projected/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-kube-api-access-dw6f5\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg\" (UID: \"d8da888f-3fd6-4d71-a5be-ad96ea3fd427\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" Nov 01 06:31:31.771949 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.771871 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg\" (UID: \"d8da888f-3fd6-4d71-a5be-ad96ea3fd427\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" Nov 01 06:31:31.772462 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.772436 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg\" (UID: \"d8da888f-3fd6-4d71-a5be-ad96ea3fd427\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" Nov 01 06:31:31.772557 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.772441 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg\" (UID: \"d8da888f-3fd6-4d71-a5be-ad96ea3fd427\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" Nov 01 06:31:31.774473 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.774457 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg\" (UID: \"d8da888f-3fd6-4d71-a5be-ad96ea3fd427\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" Nov 01 06:31:31.827942 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.827903 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw6f5\" (UniqueName: \"kubernetes.io/projected/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-kube-api-access-dw6f5\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg\" (UID: \"d8da888f-3fd6-4d71-a5be-ad96ea3fd427\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" Nov 01 06:31:31.835918 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:31.835900 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" Nov 01 06:31:32.213319 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:31:32.213294 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8da888f_3fd6_4d71_a5be_ad96ea3fd427.slice/crio-0f7c5ee6b046669b47fb489ad756f4786743426e9a39d69df8fe66e729855367 WatchSource:0}: Error finding container 0f7c5ee6b046669b47fb489ad756f4786743426e9a39d69df8fe66e729855367: Status 404 returned error can't find the container with id 0f7c5ee6b046669b47fb489ad756f4786743426e9a39d69df8fe66e729855367 Nov 01 06:31:32.214489 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:32.214463 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg"] Nov 01 06:31:33.072796 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:33.072758 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" event={"ID":"d8da888f-3fd6-4d71-a5be-ad96ea3fd427","Type":"ContainerStarted","Data":"1a464fdb826c68898673cf159a5369ca2949e59fe09970f7674ac01342c97897"} Nov 01 06:31:33.073238 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:33.072821 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" event={"ID":"d8da888f-3fd6-4d71-a5be-ad96ea3fd427","Type":"ContainerStarted","Data":"0f7c5ee6b046669b47fb489ad756f4786743426e9a39d69df8fe66e729855367"} Nov 01 06:31:33.073238 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:33.073026 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" Nov 01 06:31:33.124631 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:33.124586 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:33.188283 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:33.188231 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" podStartSLOduration=2.188217288 podStartE2EDuration="2.188217288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:31:33.185222915 +0000 UTC m=+1302.949022807" watchObservedRunningTime="2026-11-01 06:31:33.188217288 +0000 UTC m=+1302.952017150" Nov 01 06:31:34.454328 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:34.454299 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6"] Nov 01 06:31:34.548331 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:34.548299 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:36.780752 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:36.780710 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:31:37.513381 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.513350 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb"] Nov 01 06:31:37.517567 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.517551 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.552651 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.552621 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-workload-certs\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.552783 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.552657 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-istio-envoy\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.552783 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.552682 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-istio-data\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.552894 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.552853 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-workload-socket\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.552930 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.552920 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d360093e-42be-4758-9760-9fe61e233e4e-istiod-ca-cert\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.552966 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.552953 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz6j8\" (UniqueName: \"kubernetes.io/projected/d360093e-42be-4758-9760-9fe61e233e4e-kube-api-access-wz6j8\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.553043 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.553027 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d360093e-42be-4758-9760-9fe61e233e4e-istio-podinfo\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.553080 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.553070 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-credential-socket\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.553203 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.553187 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d360093e-42be-4758-9760-9fe61e233e4e-istio-token\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.569354 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.569326 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-dc6e48-openshift-default-dockercfg-6n7lp\"" Nov 01 06:31:37.625462 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.625437 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb"] Nov 01 06:31:37.654287 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.654250 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz6j8\" (UniqueName: \"kubernetes.io/projected/d360093e-42be-4758-9760-9fe61e233e4e-kube-api-access-wz6j8\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.654415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.654311 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d360093e-42be-4758-9760-9fe61e233e4e-istio-podinfo\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.654415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.654352 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-credential-socket\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.654516 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.654435 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d360093e-42be-4758-9760-9fe61e233e4e-istio-token\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.654516 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.654491 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-workload-certs\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.654615 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.654519 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-istio-envoy\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.654615 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.654556 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-istio-data\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.654956 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.654923 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-workload-socket\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.655081 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.655002 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d360093e-42be-4758-9760-9fe61e233e4e-istiod-ca-cert\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.655520 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.655500 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-credential-socket\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.655986 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.655967 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-istio-data\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.656110 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.655981 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-workload-certs\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.656216 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.656048 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-workload-socket\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.656436 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.656413 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d360093e-42be-4758-9760-9fe61e233e4e-istiod-ca-cert\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.657235 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.657213 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-istio-envoy\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.657626 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.657604 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d360093e-42be-4758-9760-9fe61e233e4e-istio-podinfo\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.710620 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.710587 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d360093e-42be-4758-9760-9fe61e233e4e-istio-token\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.710753 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.710638 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz6j8\" (UniqueName: \"kubernetes.io/projected/d360093e-42be-4758-9760-9fe61e233e4e-kube-api-access-wz6j8\") pod \"e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.832209 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.832179 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:37.832695 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:37.832461 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:38.331937 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:31:38.331908 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd360093e_42be_4758_9760_9fe61e233e4e.slice/crio-22e1e23a37944d77226b1c97314689f68d6bcc500598abce93475766ed2273e6 WatchSource:0}: Error finding container 22e1e23a37944d77226b1c97314689f68d6bcc500598abce93475766ed2273e6: Status 404 returned error can't find the container with id 22e1e23a37944d77226b1c97314689f68d6bcc500598abce93475766ed2273e6 Nov 01 06:31:38.332256 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:38.332234 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb"] Nov 01 06:31:38.334383 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:38.334352 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:31:38.334453 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:38.334414 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:31:38.334453 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:38.334443 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:31:39.119706 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:39.119675 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" event={"ID":"d360093e-42be-4758-9760-9fe61e233e4e","Type":"ContainerStarted","Data":"9acf0c21bcca3390bef90409cbe9d661b71bf5bdb3d2a226283293eca273c1be"} Nov 01 06:31:39.120194 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:39.119724 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" event={"ID":"d360093e-42be-4758-9760-9fe61e233e4e","Type":"ContainerStarted","Data":"22e1e23a37944d77226b1c97314689f68d6bcc500598abce93475766ed2273e6"} Nov 01 06:31:39.345455 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:39.345402 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" podStartSLOduration=2.345387678 podStartE2EDuration="2.345387678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:31:39.294351634 +0000 UTC m=+1309.058151493" watchObservedRunningTime="2026-11-01 06:31:39.345387678 +0000 UTC m=+1309.109187536" Nov 01 06:31:39.832310 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:39.832275 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:39.833739 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:39.833715 4385 patch_prober.go:29] interesting pod/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 01 06:31:39.833818 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:39.833770 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" podUID="d360093e-42be-4758-9760-9fe61e233e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 01 06:31:40.833478 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:40.833437 4385 patch_prober.go:29] interesting pod/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 01 06:31:40.834024 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:40.833510 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" podUID="d360093e-42be-4758-9760-9fe61e233e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 01 06:31:41.287485 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:41.287395 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:41.833604 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:41.833572 4385 patch_prober.go:29] interesting pod/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 01 06:31:41.834055 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:41.833630 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" podUID="d360093e-42be-4758-9760-9fe61e233e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 01 06:31:42.539660 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:42.539623 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:42.833168 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:42.833129 4385 patch_prober.go:29] interesting pod/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 01 06:31:42.833381 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:42.833204 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" podUID="d360093e-42be-4758-9760-9fe61e233e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 01 06:31:43.833437 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:43.833399 4385 patch_prober.go:29] interesting pod/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 01 06:31:43.833847 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:43.833458 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" podUID="d360093e-42be-4758-9760-9fe61e233e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 01 06:31:44.047416 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:44.047377 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df43ec64-gvm76"] Nov 01 06:31:44.106556 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:44.106469 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" Nov 01 06:31:44.108888 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:44.108520 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df43ec64-gvm76"] Nov 01 06:31:44.264543 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:44.264500 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp"] Nov 01 06:31:44.387372 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:44.387287 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12175cf8-da78-4493-8764-7d1f189e006d" path="/var/lib/kubelet/pods/12175cf8-da78-4493-8764-7d1f189e006d/volumes" Nov 01 06:31:45.553463 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:45.553425 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:45.643942 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:45.639299 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:45.644587 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:45.644526 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:45.648788 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:45.648759 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:31:47.555656 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:47.555623 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:49.265532 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.265477 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" podUID="68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5" containerName="payload-processing" containerID="cri-o://9b750cf3091956ac83be4cf868a274c0d3dece0d669f82429965df6e03450111" gracePeriod=25 Nov 01 06:31:49.475967 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.475750 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" podUID="ade9c35c-3890-4fca-844a-7082358cbcf6" containerName="main" containerID="cri-o://610a8725d6b24651479631a63279ed199d73056a460ccaf9df05263429e150dd" gracePeriod=45 Nov 01 06:31:49.615932 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.615893 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" Nov 01 06:31:49.708913 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.708882 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-routing-overlay-volume\") pod \"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5\" (UID: \"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5\") " Nov 01 06:31:49.709110 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.709016 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-kube-api-access-cp78x\" (UniqueName: \"kubernetes.io/projected/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-kube-api-access-cp78x\") pod \"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5\" (UID: \"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5\") " Nov 01 06:31:49.709110 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.709055 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-provider-credentials\") pod \"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5\" (UID: \"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5\") " Nov 01 06:31:49.709110 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.709096 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-plugins-config-volume\") pod \"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5\" (UID: \"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5\") " Nov 01 06:31:49.709351 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.709318 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-routing-overlay-volume" pod "68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5" (UID: "68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:31:49.709655 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.709633 4385 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-routing-overlay-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:49.709749 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.709727 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-plugins-config-volume" pod "68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5" (UID: "68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:31:49.711745 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.711717 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-provider-credentials" pod "68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5" (UID: "68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:31:49.712701 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.712680 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-kube-api-access-cp78x" pod "68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5" (UID: "68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5"). InnerVolumeSpecName "kube-api-access-cp78x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:31:49.744882 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.744857 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:31:49.811245 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.811220 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp78x\" (UniqueName: \"kubernetes.io/projected/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-kube-api-access-cp78x\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:49.811245 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.811243 4385 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-provider-credentials\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:49.811465 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.811256 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:49.912105 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.912070 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ade9c35c-3890-4fca-844a-7082358cbcf6-tls-certs\" (UniqueName: \"kubernetes.io/secret/ade9c35c-3890-4fca-844a-7082358cbcf6-tls-certs\") pod \"ade9c35c-3890-4fca-844a-7082358cbcf6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " Nov 01 06:31:49.912297 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.912188 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-dshm\") pod \"ade9c35c-3890-4fca-844a-7082358cbcf6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " Nov 01 06:31:49.912297 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.912262 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-tmp-dir\") pod \"ade9c35c-3890-4fca-844a-7082358cbcf6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " Nov 01 06:31:49.912432 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.912334 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade9c35c-3890-4fca-844a-7082358cbcf6-kube-api-access-xs2zk\" (UniqueName: \"kubernetes.io/projected/ade9c35c-3890-4fca-844a-7082358cbcf6-kube-api-access-xs2zk\") pod \"ade9c35c-3890-4fca-844a-7082358cbcf6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " Nov 01 06:31:49.912432 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.912376 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-home\" (UniqueName: \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-home\") pod \"ade9c35c-3890-4fca-844a-7082358cbcf6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " Nov 01 06:31:49.912539 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.912452 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-model-cache\") pod \"ade9c35c-3890-4fca-844a-7082358cbcf6\" (UID: \"ade9c35c-3890-4fca-844a-7082358cbcf6\") " Nov 01 06:31:49.912595 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.912528 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-tmp-dir" pod "ade9c35c-3890-4fca-844a-7082358cbcf6" (UID: "ade9c35c-3890-4fca-844a-7082358cbcf6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:31:49.912689 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.912663 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-home" pod "ade9c35c-3890-4fca-844a-7082358cbcf6" (UID: "ade9c35c-3890-4fca-844a-7082358cbcf6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:31:49.912949 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.912902 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-model-cache" pod "ade9c35c-3890-4fca-844a-7082358cbcf6" (UID: "ade9c35c-3890-4fca-844a-7082358cbcf6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:31:49.913303 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.913005 4385 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-tmp-dir\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:49.913303 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.913028 4385 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-home\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:49.914730 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.914706 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-dshm" pod "ade9c35c-3890-4fca-844a-7082358cbcf6" (UID: "ade9c35c-3890-4fca-844a-7082358cbcf6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:31:49.914880 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.914819 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade9c35c-3890-4fca-844a-7082358cbcf6-kube-api-access-xs2zk" pod "ade9c35c-3890-4fca-844a-7082358cbcf6" (UID: "ade9c35c-3890-4fca-844a-7082358cbcf6"). InnerVolumeSpecName "kube-api-access-xs2zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:31:49.915028 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:49.915006 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade9c35c-3890-4fca-844a-7082358cbcf6-tls-certs" pod "ade9c35c-3890-4fca-844a-7082358cbcf6" (UID: "ade9c35c-3890-4fca-844a-7082358cbcf6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:31:50.013666 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.013635 4385 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-dshm\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:50.013666 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.013662 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs2zk\" (UniqueName: \"kubernetes.io/projected/ade9c35c-3890-4fca-844a-7082358cbcf6-kube-api-access-xs2zk\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:50.013666 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.013674 4385 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade9c35c-3890-4fca-844a-7082358cbcf6-model-cache\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:50.014032 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.013683 4385 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade9c35c-3890-4fca-844a-7082358cbcf6-tls-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:50.232721 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.232628 4385 generic.go:356] "Generic (PLEG): container finished" podID="68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5" containerID="9b750cf3091956ac83be4cf868a274c0d3dece0d669f82429965df6e03450111" exitCode=0 Nov 01 06:31:50.232721 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.232674 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" Nov 01 06:31:50.232967 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.232724 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" event={"ID":"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5","Type":"ContainerDied","Data":"9b750cf3091956ac83be4cf868a274c0d3dece0d669f82429965df6e03450111"} Nov 01 06:31:50.232967 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.232786 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp" event={"ID":"68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5","Type":"ContainerDied","Data":"8254061a385365848f8615a1995a9b5cb1b036d18c929cdb81d8a86af42e5c7d"} Nov 01 06:31:50.232967 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.232817 4385 scope.go:122] "RemoveContainer" containerID="9b750cf3091956ac83be4cf868a274c0d3dece0d669f82429965df6e03450111" Nov 01 06:31:50.234141 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.234121 4385 generic.go:356] "Generic (PLEG): container finished" podID="ade9c35c-3890-4fca-844a-7082358cbcf6" containerID="610a8725d6b24651479631a63279ed199d73056a460ccaf9df05263429e150dd" exitCode=0 Nov 01 06:31:50.234239 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.234160 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" event={"ID":"ade9c35c-3890-4fca-844a-7082358cbcf6","Type":"ContainerDied","Data":"610a8725d6b24651479631a63279ed199d73056a460ccaf9df05263429e150dd"} Nov 01 06:31:50.234239 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.234186 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" Nov 01 06:31:50.234239 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.234192 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6" event={"ID":"ade9c35c-3890-4fca-844a-7082358cbcf6","Type":"ContainerDied","Data":"efc0c707221ab59e9810646e64b9fe1606c02df955e82287dd1a7eda3505952c"} Nov 01 06:31:50.243363 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.243348 4385 scope.go:122] "RemoveContainer" containerID="9b750cf3091956ac83be4cf868a274c0d3dece0d669f82429965df6e03450111" Nov 01 06:31:50.243636 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:50.243614 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b750cf3091956ac83be4cf868a274c0d3dece0d669f82429965df6e03450111\": container with ID starting with 9b750cf3091956ac83be4cf868a274c0d3dece0d669f82429965df6e03450111 not found: ID does not exist" containerID="9b750cf3091956ac83be4cf868a274c0d3dece0d669f82429965df6e03450111" Nov 01 06:31:50.243745 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.243642 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b750cf3091956ac83be4cf868a274c0d3dece0d669f82429965df6e03450111"} err="failed to get container status \"9b750cf3091956ac83be4cf868a274c0d3dece0d669f82429965df6e03450111\": rpc error: code = NotFound desc = could not find container \"9b750cf3091956ac83be4cf868a274c0d3dece0d669f82429965df6e03450111\": container with ID starting with 9b750cf3091956ac83be4cf868a274c0d3dece0d669f82429965df6e03450111 not found: ID does not exist" Nov 01 06:31:50.243745 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.243660 4385 scope.go:122] "RemoveContainer" containerID="610a8725d6b24651479631a63279ed199d73056a460ccaf9df05263429e150dd" Nov 01 06:31:50.253120 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.253101 4385 scope.go:122] "RemoveContainer" containerID="610a8725d6b24651479631a63279ed199d73056a460ccaf9df05263429e150dd" Nov 01 06:31:50.253399 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:50.253382 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"610a8725d6b24651479631a63279ed199d73056a460ccaf9df05263429e150dd\": container with ID starting with 610a8725d6b24651479631a63279ed199d73056a460ccaf9df05263429e150dd not found: ID does not exist" containerID="610a8725d6b24651479631a63279ed199d73056a460ccaf9df05263429e150dd" Nov 01 06:31:50.253450 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.253406 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"610a8725d6b24651479631a63279ed199d73056a460ccaf9df05263429e150dd"} err="failed to get container status \"610a8725d6b24651479631a63279ed199d73056a460ccaf9df05263429e150dd\": rpc error: code = NotFound desc = could not find container \"610a8725d6b24651479631a63279ed199d73056a460ccaf9df05263429e150dd\": container with ID starting with 610a8725d6b24651479631a63279ed199d73056a460ccaf9df05263429e150dd not found: ID does not exist" Nov 01 06:31:50.399106 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.399076 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6"] Nov 01 06:31:50.446112 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.446086 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-bd9d24/ipp-route-bd9d24-kserve-86df4c6fb9-zzkt6"] Nov 01 06:31:50.510369 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.510293 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp"] Nov 01 06:31:50.516034 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.516005 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5v6ggp"] Nov 01 06:31:50.726666 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.726638 4385 scope.go:122] "RemoveContainer" containerID="1fcb2ca0f766c825333a8cf773a3fc2379221318c2c64aeacfce98475e674435" Nov 01 06:31:50.736519 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.736502 4385 scope.go:122] "RemoveContainer" containerID="78b855e648e3ea909965e0e28e78f9318429f6feb1baab8b02406d98ab5e19fa" Nov 01 06:31:50.748117 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:50.746793 4385 scope.go:122] "RemoveContainer" containerID="d6dd8535e01722485f3cebbe71613c13906faef60a65a1184da7d84aea17499a" Nov 01 06:31:52.385369 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:52.385333 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5" path="/var/lib/kubelet/pods/68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5/volumes" Nov 01 06:31:52.385787 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:52.385762 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade9c35c-3890-4fca-844a-7082358cbcf6" path="/var/lib/kubelet/pods/ade9c35c-3890-4fca-844a-7082358cbcf6/volumes" Nov 01 06:31:52.438285 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:52.438253 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:54.233186 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.233153 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da5127df-8g48m"] Nov 01 06:31:54.233680 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.233666 4385 cpu_manager.go:414] "containerMap: removing container" podUID="ade9c35c-3890-4fca-844a-7082358cbcf6" containerName="main" Nov 01 06:31:54.233723 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.233680 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade9c35c-3890-4fca-844a-7082358cbcf6" containerName="main" Nov 01 06:31:54.233759 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.233723 4385 cpu_manager.go:414] "containerMap: removing container" podUID="68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5" containerName="payload-processing" Nov 01 06:31:54.233759 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.233728 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5" containerName="payload-processing" Nov 01 06:31:54.233827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.233810 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d1320e-82f2-4b5d-a0e3-9dd1ffe583a5" containerName="payload-processing" Nov 01 06:31:54.233827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.233821 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade9c35c-3890-4fca-844a-7082358cbcf6" containerName="main" Nov 01 06:31:54.239109 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.239089 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5127df-8g48m" Nov 01 06:31:54.264686 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.264663 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/046657d1-34a3-44d6-88e2-8c269f0c7001-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da5127df-8g48m\" (UID: \"046657d1-34a3-44d6-88e2-8c269f0c7001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5127df-8g48m" Nov 01 06:31:54.264817 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.264739 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgs98\" (UniqueName: \"kubernetes.io/projected/046657d1-34a3-44d6-88e2-8c269f0c7001-kube-api-access-kgs98\") pod \"maas-api-revoke-sub-da5127df-8g48m\" (UID: \"046657d1-34a3-44d6-88e2-8c269f0c7001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5127df-8g48m" Nov 01 06:31:54.264889 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.264871 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/046657d1-34a3-44d6-88e2-8c269f0c7001-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da5127df-8g48m\" (UID: \"046657d1-34a3-44d6-88e2-8c269f0c7001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5127df-8g48m" Nov 01 06:31:54.297916 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.297890 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\"" Nov 01 06:31:54.349558 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.349526 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da5127df-8g48m"] Nov 01 06:31:54.366004 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.365976 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/046657d1-34a3-44d6-88e2-8c269f0c7001-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da5127df-8g48m\" (UID: \"046657d1-34a3-44d6-88e2-8c269f0c7001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5127df-8g48m" Nov 01 06:31:54.366162 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.366073 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgs98\" (UniqueName: \"kubernetes.io/projected/046657d1-34a3-44d6-88e2-8c269f0c7001-kube-api-access-kgs98\") pod \"maas-api-revoke-sub-da5127df-8g48m\" (UID: \"046657d1-34a3-44d6-88e2-8c269f0c7001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5127df-8g48m" Nov 01 06:31:54.366162 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.366158 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/046657d1-34a3-44d6-88e2-8c269f0c7001-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da5127df-8g48m\" (UID: \"046657d1-34a3-44d6-88e2-8c269f0c7001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5127df-8g48m" Nov 01 06:31:54.366709 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.366690 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/046657d1-34a3-44d6-88e2-8c269f0c7001-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da5127df-8g48m\" (UID: \"046657d1-34a3-44d6-88e2-8c269f0c7001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5127df-8g48m" Nov 01 06:31:54.396953 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.396917 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 06:31:54.410047 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.410005 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/046657d1-34a3-44d6-88e2-8c269f0c7001-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da5127df-8g48m\" (UID: \"046657d1-34a3-44d6-88e2-8c269f0c7001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5127df-8g48m" Nov 01 06:31:54.435592 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.435560 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgs98\" (UniqueName: \"kubernetes.io/projected/046657d1-34a3-44d6-88e2-8c269f0c7001-kube-api-access-kgs98\") pod \"maas-api-revoke-sub-da5127df-8g48m\" (UID: \"046657d1-34a3-44d6-88e2-8c269f0c7001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5127df-8g48m" Nov 01 06:31:54.552721 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.552634 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5127df-8g48m" Nov 01 06:31:54.935062 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:31:54.935027 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046657d1_34a3_44d6_88e2_8c269f0c7001.slice/crio-2b4ce69c3cede7cb86e6179f88e940dfc85b6a549c41f7b6201eba242e389cf8 WatchSource:0}: Error finding container 2b4ce69c3cede7cb86e6179f88e940dfc85b6a549c41f7b6201eba242e389cf8: Status 404 returned error can't find the container with id 2b4ce69c3cede7cb86e6179f88e940dfc85b6a549c41f7b6201eba242e389cf8 Nov 01 06:31:54.935512 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:54.935487 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da5127df-8g48m"] Nov 01 06:31:55.272049 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:55.272014 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5127df-8g48m" event={"ID":"046657d1-34a3-44d6-88e2-8c269f0c7001","Type":"ContainerStarted","Data":"2b4ce69c3cede7cb86e6179f88e940dfc85b6a549c41f7b6201eba242e389cf8"} Nov 01 06:31:56.280331 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:56.280246 4385 generic.go:356] "Generic (PLEG): container finished" podID="046657d1-34a3-44d6-88e2-8c269f0c7001" containerID="1789b5ed4ad213344b902ddb1250ffd650b123359e3942796e0b72a2d23fe1c5" exitCode=0 Nov 01 06:31:56.280721 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:56.280334 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5127df-8g48m" event={"ID":"046657d1-34a3-44d6-88e2-8c269f0c7001","Type":"ContainerDied","Data":"1789b5ed4ad213344b902ddb1250ffd650b123359e3942796e0b72a2d23fe1c5"} Nov 01 06:31:57.447082 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:57.447053 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5127df-8g48m" Nov 01 06:31:57.505824 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:57.505794 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/046657d1-34a3-44d6-88e2-8c269f0c7001-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/046657d1-34a3-44d6-88e2-8c269f0c7001-maas-api-openshift-ca\") pod \"046657d1-34a3-44d6-88e2-8c269f0c7001\" (UID: \"046657d1-34a3-44d6-88e2-8c269f0c7001\") " Nov 01 06:31:57.506043 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:57.505910 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/046657d1-34a3-44d6-88e2-8c269f0c7001-kube-api-access-kgs98\" (UniqueName: \"kubernetes.io/projected/046657d1-34a3-44d6-88e2-8c269f0c7001-kube-api-access-kgs98\") pod \"046657d1-34a3-44d6-88e2-8c269f0c7001\" (UID: \"046657d1-34a3-44d6-88e2-8c269f0c7001\") " Nov 01 06:31:57.506043 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:57.505950 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/046657d1-34a3-44d6-88e2-8c269f0c7001-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/046657d1-34a3-44d6-88e2-8c269f0c7001-maas-api-xks-ca\") pod \"046657d1-34a3-44d6-88e2-8c269f0c7001\" (UID: \"046657d1-34a3-44d6-88e2-8c269f0c7001\") " Nov 01 06:31:57.506222 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:57.506197 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046657d1-34a3-44d6-88e2-8c269f0c7001-maas-api-openshift-ca" pod "046657d1-34a3-44d6-88e2-8c269f0c7001" (UID: "046657d1-34a3-44d6-88e2-8c269f0c7001"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:31:57.506458 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:57.506438 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/046657d1-34a3-44d6-88e2-8c269f0c7001-maas-api-openshift-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:57.508177 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:57.508156 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/046657d1-34a3-44d6-88e2-8c269f0c7001-maas-api-xks-ca" pod "046657d1-34a3-44d6-88e2-8c269f0c7001" (UID: "046657d1-34a3-44d6-88e2-8c269f0c7001"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:31:57.508305 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:57.508212 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046657d1-34a3-44d6-88e2-8c269f0c7001-kube-api-access-kgs98" pod "046657d1-34a3-44d6-88e2-8c269f0c7001" (UID: "046657d1-34a3-44d6-88e2-8c269f0c7001"). InnerVolumeSpecName "kube-api-access-kgs98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:31:57.607661 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:57.607630 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgs98\" (UniqueName: \"kubernetes.io/projected/046657d1-34a3-44d6-88e2-8c269f0c7001-kube-api-access-kgs98\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:57.607661 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:57.607655 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/046657d1-34a3-44d6-88e2-8c269f0c7001-maas-api-xks-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:31:58.298330 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.298295 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5127df-8g48m" Nov 01 06:31:58.298513 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.298331 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5127df-8g48m" event={"ID":"046657d1-34a3-44d6-88e2-8c269f0c7001","Type":"ContainerDied","Data":"2b4ce69c3cede7cb86e6179f88e940dfc85b6a549c41f7b6201eba242e389cf8"} Nov 01 06:31:58.298513 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.298385 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b4ce69c3cede7cb86e6179f88e940dfc85b6a549c41f7b6201eba242e389cf8" Nov 01 06:31:58.475278 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.475247 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh"] Nov 01 06:31:58.475816 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.475800 4385 cpu_manager.go:414] "containerMap: removing container" podUID="046657d1-34a3-44d6-88e2-8c269f0c7001" containerName="revoke-keys" Nov 01 06:31:58.475903 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.475817 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="046657d1-34a3-44d6-88e2-8c269f0c7001" containerName="revoke-keys" Nov 01 06:31:58.475972 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.475961 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="046657d1-34a3-44d6-88e2-8c269f0c7001" containerName="revoke-keys" Nov 01 06:31:58.479628 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.479609 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" Nov 01 06:31:58.518053 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.518017 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a800ad90-b099-4484-bbf5-0c1ce2802c73-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh\" (UID: \"a800ad90-b099-4484-bbf5-0c1ce2802c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" Nov 01 06:31:58.518212 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.518074 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a800ad90-b099-4484-bbf5-0c1ce2802c73-metrics-certs\") pod \"maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh\" (UID: \"a800ad90-b099-4484-bbf5-0c1ce2802c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" Nov 01 06:31:58.518212 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.518188 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a800ad90-b099-4484-bbf5-0c1ce2802c73-maas-api-tls\") pod \"maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh\" (UID: \"a800ad90-b099-4484-bbf5-0c1ce2802c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" Nov 01 06:31:58.518297 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.518260 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5ngk\" (UniqueName: \"kubernetes.io/projected/a800ad90-b099-4484-bbf5-0c1ce2802c73-kube-api-access-s5ngk\") pod \"maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh\" (UID: \"a800ad90-b099-4484-bbf5-0c1ce2802c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" Nov 01 06:31:58.531746 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.531719 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-25719d-serving-cert\"" Nov 01 06:31:58.531908 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.531858 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-csmhq\"" Nov 01 06:31:58.582147 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.582115 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 06:31:58.587200 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.587170 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh"] Nov 01 06:31:58.619029 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.619003 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5ngk\" (UniqueName: \"kubernetes.io/projected/a800ad90-b099-4484-bbf5-0c1ce2802c73-kube-api-access-s5ngk\") pod \"maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh\" (UID: \"a800ad90-b099-4484-bbf5-0c1ce2802c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" Nov 01 06:31:58.619202 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.619108 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a800ad90-b099-4484-bbf5-0c1ce2802c73-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh\" (UID: \"a800ad90-b099-4484-bbf5-0c1ce2802c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" Nov 01 06:31:58.619202 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.619141 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a800ad90-b099-4484-bbf5-0c1ce2802c73-metrics-certs\") pod \"maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh\" (UID: \"a800ad90-b099-4484-bbf5-0c1ce2802c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" Nov 01 06:31:58.619202 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.619181 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a800ad90-b099-4484-bbf5-0c1ce2802c73-maas-api-tls\") pod \"maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh\" (UID: \"a800ad90-b099-4484-bbf5-0c1ce2802c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" Nov 01 06:31:58.619389 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:58.619300 4385 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-serving-cert: secret "maas-api-e2e-extmod-a-25719d-serving-cert" not found Nov 01 06:31:58.619389 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:31:58.619346 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a800ad90-b099-4484-bbf5-0c1ce2802c73-maas-api-tls podName:a800ad90-b099-4484-bbf5-0c1ce2802c73 nodeName:}" failed. No retries permitted until 2026-11-01 06:31:59.119331819 +0000 UTC m=+1328.883131655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a800ad90-b099-4484-bbf5-0c1ce2802c73-maas-api-tls") pod "maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" (UID: "a800ad90-b099-4484-bbf5-0c1ce2802c73") : secret "maas-api-e2e-extmod-a-25719d-serving-cert" not found Nov 01 06:31:58.620197 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.620172 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a800ad90-b099-4484-bbf5-0c1ce2802c73-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh\" (UID: \"a800ad90-b099-4484-bbf5-0c1ce2802c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" Nov 01 06:31:58.622296 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.622271 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a800ad90-b099-4484-bbf5-0c1ce2802c73-metrics-certs\") pod \"maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh\" (UID: \"a800ad90-b099-4484-bbf5-0c1ce2802c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" Nov 01 06:31:58.675889 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.675860 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5ngk\" (UniqueName: \"kubernetes.io/projected/a800ad90-b099-4484-bbf5-0c1ce2802c73-kube-api-access-s5ngk\") pod \"maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh\" (UID: \"a800ad90-b099-4484-bbf5-0c1ce2802c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" Nov 01 06:31:58.680050 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.680022 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:58.868977 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:58.868889 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:59.018671 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:59.018627 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:31:59.045092 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:59.045064 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2491bf03-746jt"] Nov 01 06:31:59.048719 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:59.048697 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2491bf03-746jt"] Nov 01 06:31:59.127493 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:59.127399 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a800ad90-b099-4484-bbf5-0c1ce2802c73-maas-api-tls\") pod \"maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh\" (UID: \"a800ad90-b099-4484-bbf5-0c1ce2802c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" Nov 01 06:31:59.130147 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:59.130124 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a800ad90-b099-4484-bbf5-0c1ce2802c73-maas-api-tls\") pod \"maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh\" (UID: \"a800ad90-b099-4484-bbf5-0c1ce2802c73\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" Nov 01 06:31:59.398250 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:59.398160 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" Nov 01 06:31:59.784940 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:59.784900 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh"] Nov 01 06:31:59.786103 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:31:59.786063 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda800ad90_b099_4484_bbf5_0c1ce2802c73.slice/crio-847e8a0a64c1b21e5f477e9e1bc3a40588be18c31b6bb21e760a0e8ee58b6377 WatchSource:0}: Error finding container 847e8a0a64c1b21e5f477e9e1bc3a40588be18c31b6bb21e760a0e8ee58b6377: Status 404 returned error can't find the container with id 847e8a0a64c1b21e5f477e9e1bc3a40588be18c31b6bb21e760a0e8ee58b6377 Nov 01 06:31:59.787968 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:31:59.787951 4385 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:32:00.316362 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:00.316325 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" event={"ID":"a800ad90-b099-4484-bbf5-0c1ce2802c73","Type":"ContainerStarted","Data":"847e8a0a64c1b21e5f477e9e1bc3a40588be18c31b6bb21e760a0e8ee58b6377"} Nov 01 06:32:00.386931 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:00.386893 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09caa300-7be1-4c82-8a88-8b4abc2cba1c" path="/var/lib/kubelet/pods/09caa300-7be1-4c82-8a88-8b4abc2cba1c/volumes" Nov 01 06:32:02.343882 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:02.343798 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" event={"ID":"a800ad90-b099-4484-bbf5-0c1ce2802c73","Type":"ContainerStarted","Data":"0b6349d6c8c247088c84765d0b0d7cd3b5758fccd53eb11cafc09622ff6fc0b2"} Nov 01 06:32:02.344372 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:02.344056 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" Nov 01 06:32:02.460464 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:02.460413 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" podStartSLOduration=2.394282263 podStartE2EDuration="4.460397123s" totalImagesPullingTime="2.06611486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:31:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:32:02.456925221 +0000 UTC m=+1332.220725113" watchObservedRunningTime="2026-11-01 06:32:02.460397123 +0000 UTC m=+1332.224196980" Nov 01 06:32:03.542436 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:03.542403 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:06.782633 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:32:06.782594 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:32:07.285094 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:07.285053 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:07.443450 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:07.443403 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:07.936171 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:07.936142 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:08.365059 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:08.365033 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" Nov 01 06:32:08.523776 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:08.523733 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:09.834130 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:09.834099 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:11.234882 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:11.234843 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:11.447971 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:11.447940 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:13.628414 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:13.628374 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:13.833215 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:13.833183 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:13.972866 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:13.972769 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:14.726147 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.726109 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99"] Nov 01 06:32:14.732112 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.732090 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.784202 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.784169 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-56ae52d5-openshift-default-dockercfg-bhfct\"" Nov 01 06:32:14.844109 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.844075 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-credential-socket\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.883742 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.855561 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:14.883742 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.867127 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99"] Nov 01 06:32:14.883742 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.868245 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-workload-socket\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.883742 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.868347 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d92af192-040f-459a-aa4a-bc6e90460eb6-istiod-ca-cert\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.883742 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.868441 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-data\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.883742 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.868479 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-podinfo\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.883742 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.868567 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6rdp\" (UniqueName: \"kubernetes.io/projected/d92af192-040f-459a-aa4a-bc6e90460eb6-kube-api-access-x6rdp\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.883742 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.868636 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-envoy\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.883742 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.868742 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-token\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.883742 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.868794 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-workload-certs\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.974861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.973640 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-token\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.974861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.973708 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-workload-certs\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.974861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.973792 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-credential-socket\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.974861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.973851 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-workload-socket\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.974861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.973924 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d92af192-040f-459a-aa4a-bc6e90460eb6-istiod-ca-cert\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.974861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.973953 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-data\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.974861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.973984 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-podinfo\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.974861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.974025 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6rdp\" (UniqueName: \"kubernetes.io/projected/d92af192-040f-459a-aa4a-bc6e90460eb6-kube-api-access-x6rdp\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.974861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.974090 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-envoy\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.989914 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.980622 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-workload-socket\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.989914 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.981347 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-workload-certs\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.989914 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.981584 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-credential-socket\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.989914 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.981869 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-data\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.989914 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.982536 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d92af192-040f-459a-aa4a-bc6e90460eb6-istiod-ca-cert\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:14.989914 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:14.985897 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-podinfo\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:15.026653 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:15.026627 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-envoy\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:15.038784 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:15.038204 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-token\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:15.040626 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:15.040595 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6rdp\" (UniqueName: \"kubernetes.io/projected/d92af192-040f-459a-aa4a-bc6e90460eb6-kube-api-access-x6rdp\") pod \"e2e-ait-56ae52d5-openshift-default-796558d487-6gp99\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:15.046271 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:15.046246 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:15.268292 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:15.268182 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:15.531387 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:15.531362 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99"] Nov 01 06:32:15.532983 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:32:15.532953 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd92af192_040f_459a_aa4a_bc6e90460eb6.slice/crio-3e126b674b21e1ef0bcf1f9221eb14983b93bd45f97f11206f322a86dcff68f0 WatchSource:0}: Error finding container 3e126b674b21e1ef0bcf1f9221eb14983b93bd45f97f11206f322a86dcff68f0: Status 404 returned error can't find the container with id 3e126b674b21e1ef0bcf1f9221eb14983b93bd45f97f11206f322a86dcff68f0 Nov 01 06:32:15.536109 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:15.536078 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:32:15.536196 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:15.536144 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:32:15.536196 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:15.536173 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:32:16.519857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:16.519535 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" event={"ID":"d92af192-040f-459a-aa4a-bc6e90460eb6","Type":"ContainerStarted","Data":"684c9efd19634e800134e0a14edc61b0c021ec9da0cc76c681febd56589edadf"} Nov 01 06:32:16.519857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:16.519632 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" event={"ID":"d92af192-040f-459a-aa4a-bc6e90460eb6","Type":"ContainerStarted","Data":"3e126b674b21e1ef0bcf1f9221eb14983b93bd45f97f11206f322a86dcff68f0"} Nov 01 06:32:16.693292 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:16.693224 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" podStartSLOduration=2.69320441 podStartE2EDuration="2.69320441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:32:16.642291664 +0000 UTC m=+1346.406091525" watchObservedRunningTime="2026-11-01 06:32:16.69320441 +0000 UTC m=+1346.457004267" Nov 01 06:32:17.047111 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:17.047075 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:17.058874 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:17.056929 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:17.395679 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:17.395646 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:17.526398 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:17.526366 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:17.527650 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:17.527628 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:32:18.724575 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:18.724543 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:18.953846 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:18.953796 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:19.153931 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:19.153888 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:19.314125 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:19.314091 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:21.980828 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:21.980791 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:22.178891 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.178857 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:22.537039 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.537005 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2"] Nov 01 06:32:22.544424 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.544386 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2" Nov 01 06:32:22.625101 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.621809 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6931d58-6876-4fe4-9f49-f1edba425b49-plugins-config-volume\") pod \"payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2\" (UID: \"d6931d58-6876-4fe4-9f49-f1edba425b49\") " pod="openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2" Nov 01 06:32:22.625101 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.621917 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbxbk\" (UniqueName: \"kubernetes.io/projected/d6931d58-6876-4fe4-9f49-f1edba425b49-kube-api-access-kbxbk\") pod \"payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2\" (UID: \"d6931d58-6876-4fe4-9f49-f1edba425b49\") " pod="openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2" Nov 01 06:32:22.645471 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.645419 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2"] Nov 01 06:32:22.659131 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.659108 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-56ae52d5-dockercfg-ncsd5\"" Nov 01 06:32:22.659725 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.659707 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-56ae52d5\"" Nov 01 06:32:22.723104 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.723066 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6931d58-6876-4fe4-9f49-f1edba425b49-plugins-config-volume\") pod \"payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2\" (UID: \"d6931d58-6876-4fe4-9f49-f1edba425b49\") " pod="openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2" Nov 01 06:32:22.723382 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.723356 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbxbk\" (UniqueName: \"kubernetes.io/projected/d6931d58-6876-4fe4-9f49-f1edba425b49-kube-api-access-kbxbk\") pod \"payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2\" (UID: \"d6931d58-6876-4fe4-9f49-f1edba425b49\") " pod="openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2" Nov 01 06:32:22.723858 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.723815 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6931d58-6876-4fe4-9f49-f1edba425b49-plugins-config-volume\") pod \"payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2\" (UID: \"d6931d58-6876-4fe4-9f49-f1edba425b49\") " pod="openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2" Nov 01 06:32:22.729404 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.729382 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr"] Nov 01 06:32:22.734600 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.734582 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr" Nov 01 06:32:22.779452 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.779422 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbxbk\" (UniqueName: \"kubernetes.io/projected/d6931d58-6876-4fe4-9f49-f1edba425b49-kube-api-access-kbxbk\") pod \"payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2\" (UID: \"d6931d58-6876-4fe4-9f49-f1edba425b49\") " pod="openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2" Nov 01 06:32:22.787489 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.787419 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr"] Nov 01 06:32:22.824357 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.824321 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gldnw\" (UniqueName: \"kubernetes.io/projected/62a2a248-71ad-4331-9b3b-afea7852687d-kube-api-access-gldnw\") pod \"payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr\" (UID: \"62a2a248-71ad-4331-9b3b-afea7852687d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr" Nov 01 06:32:22.824520 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.824376 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a2a248-71ad-4331-9b3b-afea7852687d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr\" (UID: \"62a2a248-71ad-4331-9b3b-afea7852687d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr" Nov 01 06:32:22.925214 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.925183 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gldnw\" (UniqueName: \"kubernetes.io/projected/62a2a248-71ad-4331-9b3b-afea7852687d-kube-api-access-gldnw\") pod \"payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr\" (UID: \"62a2a248-71ad-4331-9b3b-afea7852687d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr" Nov 01 06:32:22.925400 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.925220 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a2a248-71ad-4331-9b3b-afea7852687d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr\" (UID: \"62a2a248-71ad-4331-9b3b-afea7852687d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr" Nov 01 06:32:22.925937 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.925921 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a2a248-71ad-4331-9b3b-afea7852687d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr\" (UID: \"62a2a248-71ad-4331-9b3b-afea7852687d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr" Nov 01 06:32:22.964331 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.964301 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2" Nov 01 06:32:22.981486 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:22.981459 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gldnw\" (UniqueName: \"kubernetes.io/projected/62a2a248-71ad-4331-9b3b-afea7852687d-kube-api-access-gldnw\") pod \"payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr\" (UID: \"62a2a248-71ad-4331-9b3b-afea7852687d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr" Nov 01 06:32:23.050681 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:23.050350 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr" Nov 01 06:32:23.150367 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:23.150283 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:23.345301 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:32:23.345273 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6931d58_6876_4fe4_9f49_f1edba425b49.slice/crio-84c73976268aad55edc81e064b3beadec078d9fed0eb50af95fb3f9486e4c353 WatchSource:0}: Error finding container 84c73976268aad55edc81e064b3beadec078d9fed0eb50af95fb3f9486e4c353: Status 404 returned error can't find the container with id 84c73976268aad55edc81e064b3beadec078d9fed0eb50af95fb3f9486e4c353 Nov 01 06:32:23.345732 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:23.345702 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2"] Nov 01 06:32:23.453823 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:23.453798 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr"] Nov 01 06:32:23.454683 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:32:23.454660 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a2a248_71ad_4331_9b3b_afea7852687d.slice/crio-46a9178f674c116bee9b9d4eb410b7dfced7b4e0146b75e6f2ae2c9fe487f18b WatchSource:0}: Error finding container 46a9178f674c116bee9b9d4eb410b7dfced7b4e0146b75e6f2ae2c9fe487f18b: Status 404 returned error can't find the container with id 46a9178f674c116bee9b9d4eb410b7dfced7b4e0146b75e6f2ae2c9fe487f18b Nov 01 06:32:23.652423 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:23.652327 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr" event={"ID":"62a2a248-71ad-4331-9b3b-afea7852687d","Type":"ContainerStarted","Data":"66edd178beb0522627137586498727270e453deaa5972c42f5e944b55c6e7094"} Nov 01 06:32:23.652423 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:23.652405 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr" Nov 01 06:32:23.652618 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:23.652432 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr" event={"ID":"62a2a248-71ad-4331-9b3b-afea7852687d","Type":"ContainerStarted","Data":"46a9178f674c116bee9b9d4eb410b7dfced7b4e0146b75e6f2ae2c9fe487f18b"} Nov 01 06:32:23.654180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:23.654151 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2" event={"ID":"d6931d58-6876-4fe4-9f49-f1edba425b49","Type":"ContainerStarted","Data":"141847dda50bd850a0cfc16d1ee6df08e93ca889b63647be442d356ffdff474b"} Nov 01 06:32:23.654326 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:23.654195 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2" event={"ID":"d6931d58-6876-4fe4-9f49-f1edba425b49","Type":"ContainerStarted","Data":"84c73976268aad55edc81e064b3beadec078d9fed0eb50af95fb3f9486e4c353"} Nov 01 06:32:23.654326 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:23.654245 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2" Nov 01 06:32:23.767966 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:23.767917 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr" podStartSLOduration=1.767901545 podStartE2EDuration="1.767901545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:32:23.764036835 +0000 UTC m=+1353.527836694" watchObservedRunningTime="2026-11-01 06:32:23.767901545 +0000 UTC m=+1353.531701402" Nov 01 06:32:26.015748 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:26.015707 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:30.904629 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:30.904591 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:34.664896 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:34.664863 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2" Nov 01 06:32:34.665579 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:34.665553 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr" Nov 01 06:32:34.779111 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:34.779066 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2" podStartSLOduration=12.779051969 podStartE2EDuration="12.779051969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:32:23.880073078 +0000 UTC m=+1353.643872958" watchObservedRunningTime="2026-11-01 06:32:34.779051969 +0000 UTC m=+1364.542851862" Nov 01 06:32:35.559644 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:35.559615 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:36.780565 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:32:36.780528 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:32:40.266569 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:40.266534 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:41.617643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:41.617607 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:44.933759 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:44.933723 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:49.014758 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:49.014715 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:49.518907 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:49.515573 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:49.657617 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:49.657582 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:50.784603 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:50.784572 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:50.811698 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:50.811669 4385 scope.go:122] "RemoveContainer" containerID="6460215845cc1532d8c8c0f84ad10f8de8f79e4139aeb288d5a7e9a09f14166f" Nov 01 06:32:50.825475 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:50.825454 4385 scope.go:122] "RemoveContainer" containerID="c7d7a46bab1ec474aa8acf44fce2f8a62db47ef44f72ab22b6f0d362c4ab7d93" Nov 01 06:32:50.836395 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:50.836371 4385 scope.go:122] "RemoveContainer" containerID="eac34d58d932c859e3d4359cff37e9bc0edc9ec050b40c93ff97f9360d3825f9" Nov 01 06:32:50.850154 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:50.850009 4385 scope.go:122] "RemoveContainer" containerID="13b2cc213c4a7619e34edde04b3f3328818b3d51e22326dfc26ad45f96c7431a" Nov 01 06:32:51.330801 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:51.330768 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:55.903571 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:55.903530 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:56.117550 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:56.117510 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:56.486203 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:56.486159 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:56.637730 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:56.637696 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:32:58.848966 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:32:58.848933 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:01.019889 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:01.019856 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:01.420502 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:01.420465 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:01.978680 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:01.978643 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:02.726126 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:02.726092 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:03.184147 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.184110 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq"] Nov 01 06:33:03.190343 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.190317 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.304858 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.295271 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-1ff1dc-openshift-default-dockercfg-bz8zr\"" Nov 01 06:33:03.305705 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.305672 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-data\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.305890 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.305761 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-podinfo\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.305890 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.305856 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-credential-socket\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.306018 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.305934 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttxx9\" (UniqueName: \"kubernetes.io/projected/3c777717-c462-4ba3-bdc3-7b423808eb9b-kube-api-access-ttxx9\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.306018 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.305968 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-token\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.306018 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.305994 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c777717-c462-4ba3-bdc3-7b423808eb9b-istiod-ca-cert\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.306188 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.306052 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-workload-certs\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.306188 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.306083 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-workload-socket\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.306188 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.306158 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-envoy\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.311081 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.311033 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq"] Nov 01 06:33:03.313910 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.313873 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:03.408796 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.408759 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-podinfo\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.409116 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.409097 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-credential-socket\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.409308 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.409290 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttxx9\" (UniqueName: \"kubernetes.io/projected/3c777717-c462-4ba3-bdc3-7b423808eb9b-kube-api-access-ttxx9\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.409453 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.409439 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-token\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.409549 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.409537 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c777717-c462-4ba3-bdc3-7b423808eb9b-istiod-ca-cert\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.409683 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.409671 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-workload-certs\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.409780 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.409767 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-workload-socket\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.409955 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.409943 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-envoy\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.410102 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.410089 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-data\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.410645 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.410625 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-data\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.413366 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.413343 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-credential-socket\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.415862 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.413915 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-workload-socket\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.416054 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.414502 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-workload-certs\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.416168 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.414702 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c777717-c462-4ba3-bdc3-7b423808eb9b-istiod-ca-cert\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.417720 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.417697 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-envoy\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.419024 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.418998 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-podinfo\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.470420 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.470346 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-token\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.470587 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.470491 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttxx9\" (UniqueName: \"kubernetes.io/projected/3c777717-c462-4ba3-bdc3-7b423808eb9b-kube-api-access-ttxx9\") pod \"e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.528388 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.528347 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:03.711566 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.711512 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:03.984772 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.984743 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq"] Nov 01 06:33:03.985251 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:33:03.985182 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c777717_c462_4ba3_bdc3_7b423808eb9b.slice/crio-c2f0cee76868e0efa1e888a7ec2806fc37167254046534b8959fd5af41b94f73 WatchSource:0}: Error finding container c2f0cee76868e0efa1e888a7ec2806fc37167254046534b8959fd5af41b94f73: Status 404 returned error can't find the container with id c2f0cee76868e0efa1e888a7ec2806fc37167254046534b8959fd5af41b94f73 Nov 01 06:33:03.988353 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.988314 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:33:03.988476 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.988398 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:33:03.988476 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:03.988438 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:33:04.983641 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:04.983601 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" event={"ID":"3c777717-c462-4ba3-bdc3-7b423808eb9b","Type":"ContainerStarted","Data":"34276e826555c578a861f2d96c0bc61a9b4851851610ecfa40002414df7d95ce"} Nov 01 06:33:04.983868 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:04.983662 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" event={"ID":"3c777717-c462-4ba3-bdc3-7b423808eb9b","Type":"ContainerStarted","Data":"c2f0cee76868e0efa1e888a7ec2806fc37167254046534b8959fd5af41b94f73"} Nov 01 06:33:05.149861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:05.149798 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" podStartSLOduration=2.149782628 podStartE2EDuration="2.149782628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:33:05.147455029 +0000 UTC m=+1394.911254885" watchObservedRunningTime="2026-11-01 06:33:05.149782628 +0000 UTC m=+1394.913582486" Nov 01 06:33:05.529076 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:05.529039 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:05.535190 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:05.535162 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:05.723213 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:05.723173 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:05.991851 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:05.991811 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:05.993694 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:05.993240 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:33:06.781161 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:33:06.781122 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:33:07.946880 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:07.946800 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:08.149032 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:08.148992 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:10.693414 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:10.693385 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:10.858432 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:10.858394 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:11.003982 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:11.003864 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:11.149044 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:11.149009 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:11.315614 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:11.315516 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:11.693555 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:11.693524 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw"] Nov 01 06:33:11.777502 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:11.777460 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw" Nov 01 06:33:11.791164 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:11.791133 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw"] Nov 01 06:33:11.817094 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:11.817065 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw\" (UID: \"0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw" Nov 01 06:33:11.817264 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:11.817131 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkjc5\" (UniqueName: \"kubernetes.io/projected/0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b-kube-api-access-kkjc5\") pod \"payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw\" (UID: \"0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw" Nov 01 06:33:11.831017 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:11.830994 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-1ff1dc\"" Nov 01 06:33:11.879929 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:11.879892 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-1ff1dc-dockercfg-4d2k7\"" Nov 01 06:33:11.918921 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:11.918874 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw\" (UID: \"0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw" Nov 01 06:33:11.919112 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:11.918980 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkjc5\" (UniqueName: \"kubernetes.io/projected/0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b-kube-api-access-kkjc5\") pod \"payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw\" (UID: \"0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw" Nov 01 06:33:11.919604 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:11.919585 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw\" (UID: \"0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw" Nov 01 06:33:11.976041 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:11.975951 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkjc5\" (UniqueName: \"kubernetes.io/projected/0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b-kube-api-access-kkjc5\") pod \"payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw\" (UID: \"0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw" Nov 01 06:33:12.107367 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:12.107333 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw" Nov 01 06:33:12.490909 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:12.490878 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw"] Nov 01 06:33:12.491193 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:33:12.491171 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dad47cc_7b31_4fd7_ab7a_deaa3cb9e54b.slice/crio-c26028f275879a7f4d8a31ecafbb17721cfd3dd9f0dd8c1ccf37f0531c51662b WatchSource:0}: Error finding container c26028f275879a7f4d8a31ecafbb17721cfd3dd9f0dd8c1ccf37f0531c51662b: Status 404 returned error can't find the container with id c26028f275879a7f4d8a31ecafbb17721cfd3dd9f0dd8c1ccf37f0531c51662b Nov 01 06:33:13.043196 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:13.043162 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw" event={"ID":"0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b","Type":"ContainerStarted","Data":"b718bc68779a3e98187f9a7c837808bfe4981f59025760137bafa47d7ba4beb5"} Nov 01 06:33:13.043711 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:13.043224 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw" event={"ID":"0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b","Type":"ContainerStarted","Data":"c26028f275879a7f4d8a31ecafbb17721cfd3dd9f0dd8c1ccf37f0531c51662b"} Nov 01 06:33:13.043711 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:13.043459 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw" Nov 01 06:33:13.157412 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:13.157367 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw" podStartSLOduration=2.157354255 podStartE2EDuration="2.157354255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:33:13.156129775 +0000 UTC m=+1402.919929631" watchObservedRunningTime="2026-11-01 06:33:13.157354255 +0000 UTC m=+1402.921154120" Nov 01 06:33:15.149052 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:15.149019 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:15.648964 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:15.648926 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:15.855187 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:15.855151 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:16.911950 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:16.911911 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:21.738096 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:21.738065 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:22.347006 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:22.346977 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:22.543562 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:22.543509 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:23.047001 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:23.046971 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c96ff431-l79th"] Nov 01 06:33:23.096620 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:23.096579 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c96ff431-l79th"] Nov 01 06:33:23.860137 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:23.860099 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:24.053464 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:24.053435 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw" Nov 01 06:33:24.348197 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:24.348164 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:24.385506 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:24.385473 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beeb8104-b759-478c-8529-f83a49b5c541" path="/var/lib/kubelet/pods/beeb8104-b759-478c-8529-f83a49b5c541/volumes" Nov 01 06:33:25.260885 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:25.260818 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:25.424362 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:25.424318 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:26.644074 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:26.644035 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:31.432368 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:31.432331 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:33.568746 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:33.568704 4385 patch_prober.go:29] interesting pod/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:33:33.569236 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:33.568790 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" podUID="d92af192-040f-459a-aa4a-bc6e90460eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:33:35.313003 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:35.312965 4385 generic.go:356] "Generic (PLEG): container finished" podID="d92af192-040f-459a-aa4a-bc6e90460eb6" containerID="684c9efd19634e800134e0a14edc61b0c021ec9da0cc76c681febd56589edadf" exitCode=0 Nov 01 06:33:35.313492 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:35.313033 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" event={"ID":"d92af192-040f-459a-aa4a-bc6e90460eb6","Type":"ContainerDied","Data":"684c9efd19634e800134e0a14edc61b0c021ec9da0cc76c681febd56589edadf"} Nov 01 06:33:35.313492 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:35.313479 4385 scope.go:122] "RemoveContainer" containerID="684c9efd19634e800134e0a14edc61b0c021ec9da0cc76c681febd56589edadf" Nov 01 06:33:35.315258 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:35.315217 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:33:35.315321 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:35.315310 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:33:35.315376 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:35.315360 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:33:36.131573 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:36.131535 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:36.324619 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:36.324587 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" event={"ID":"d92af192-040f-459a-aa4a-bc6e90460eb6","Type":"ContainerStarted","Data":"99a9032ab0b065d994e134d2e686d17ebfa13a60a0c4744b895bb7aac4b665d7"} Nov 01 06:33:36.809201 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:33:36.809142 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:33:37.046658 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:37.046621 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:33:37.046877 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:37.046852 4385 patch_prober.go:29] interesting pod/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 01 06:33:37.046933 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:37.046902 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" podUID="d92af192-040f-459a-aa4a-bc6e90460eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 01 06:33:38.047183 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:38.047155 4385 patch_prober.go:29] interesting pod/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 01 06:33:38.047561 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:38.047225 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" podUID="d92af192-040f-459a-aa4a-bc6e90460eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 01 06:33:39.046757 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:39.046716 4385 patch_prober.go:29] interesting pod/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 01 06:33:39.046943 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:39.046786 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" podUID="d92af192-040f-459a-aa4a-bc6e90460eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 01 06:33:40.047171 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:40.047134 4385 patch_prober.go:29] interesting pod/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 01 06:33:40.047775 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:40.047742 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" podUID="d92af192-040f-459a-aa4a-bc6e90460eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 01 06:33:40.838828 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:40.838773 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:41.046642 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:41.046611 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6309d2cc-9p2c7"] Nov 01 06:33:41.046989 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:41.046954 4385 patch_prober.go:29] interesting pod/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 01 06:33:41.047149 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:41.046999 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" podUID="d92af192-040f-459a-aa4a-bc6e90460eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 01 06:33:41.117189 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:41.117105 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6309d2cc-9p2c7"] Nov 01 06:33:41.945011 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:41.944969 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:42.052105 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:42.052074 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:33:42.052494 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:42.052477 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:33:42.053148 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:42.053126 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:33:42.386714 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:42.386674 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="139e0fdf-2af1-473f-8203-7f6ec40c2814" path="/var/lib/kubelet/pods/139e0fdf-2af1-473f-8203-7f6ec40c2814/volumes" Nov 01 06:33:45.045266 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:45.045230 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495f7364-fdhmn"] Nov 01 06:33:45.094735 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:45.094702 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495f7364-fdhmn"] Nov 01 06:33:45.564902 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:45.564862 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:46.385319 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:46.385283 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7cf5df4-dd07-4bb3-a7dd-08215f5643f5" path="/var/lib/kubelet/pods/e7cf5df4-dd07-4bb3-a7dd-08215f5643f5/volumes" Nov 01 06:33:50.230145 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:50.230111 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:50.897186 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:50.897156 4385 scope.go:122] "RemoveContainer" containerID="5c596f1a9c8cba5128fa78bbb1350e98dd6dda819bdc14e07da1288d5a5347ce" Nov 01 06:33:54.920563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:54.920534 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:59.570603 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:59.570565 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:59.730670 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:59.730634 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:33:59.872216 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:33:59.872182 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:04.508801 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:04.508764 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:04.654184 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:04.654135 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:05.407237 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.407208 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs"] Nov 01 06:34:05.423164 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.423114 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.470877 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.470810 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.471065 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.470960 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ed21f8-2d17-47c1-9257-1701c59082f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.471065 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.471022 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.471190 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.471103 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.471190 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.471157 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.471279 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.471270 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.471315 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.471295 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.471363 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.471324 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.471405 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.471370 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8s44\" (UniqueName: \"kubernetes.io/projected/01ed21f8-2d17-47c1-9257-1701c59082f0-kube-api-access-r8s44\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.581959 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.581919 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.581959 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.581974 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.582484 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.582008 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8s44\" (UniqueName: \"kubernetes.io/projected/01ed21f8-2d17-47c1-9257-1701c59082f0-kube-api-access-r8s44\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.582484 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.582129 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.582484 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.582214 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ed21f8-2d17-47c1-9257-1701c59082f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.582484 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.582254 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.582484 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.582287 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.582484 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.582318 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.582484 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.582385 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.582814 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.582788 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.583399 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.583367 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ed21f8-2d17-47c1-9257-1701c59082f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.583530 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.583479 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.583807 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.583781 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.583953 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.583868 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.586195 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.586174 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.586482 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.586459 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.626782 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.626742 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs"] Nov 01 06:34:05.640424 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.640397 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.640424 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.640415 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8s44\" (UniqueName: \"kubernetes.io/projected/01ed21f8-2d17-47c1-9257-1701c59082f0-kube-api-access-r8s44\") pod \"maas-default-gateway-openshift-default-7d9897b54c-c69zs\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:05.739277 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:05.739186 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:06.171167 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:34:06.171139 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ed21f8_2d17_47c1_9257_1701c59082f0.slice/crio-3b11b808f4475347ea5521f748468644ee6991b8f0e1d8af31abcc0123137873 WatchSource:0}: Error finding container 3b11b808f4475347ea5521f748468644ee6991b8f0e1d8af31abcc0123137873: Status 404 returned error can't find the container with id 3b11b808f4475347ea5521f748468644ee6991b8f0e1d8af31abcc0123137873 Nov 01 06:34:06.171686 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:06.171657 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs"] Nov 01 06:34:06.173553 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:06.173527 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:34:06.173603 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:06.173590 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:34:06.173642 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:06.173620 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:34:06.596796 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:06.596761 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" event={"ID":"01ed21f8-2d17-47c1-9257-1701c59082f0","Type":"ContainerStarted","Data":"6b59c3388afdf0cbe6ee101733d58e10a99236a9fe1e38ab0a878a50b7f1da0b"} Nov 01 06:34:06.597201 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:06.596821 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" event={"ID":"01ed21f8-2d17-47c1-9257-1701c59082f0","Type":"ContainerStarted","Data":"3b11b808f4475347ea5521f748468644ee6991b8f0e1d8af31abcc0123137873"} Nov 01 06:34:06.739853 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:06.739405 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:06.762958 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:06.762910 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" podStartSLOduration=1.762894101 podStartE2EDuration="1.762894101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:34:06.762505636 +0000 UTC m=+1456.526305494" watchObservedRunningTime="2026-11-01 06:34:06.762894101 +0000 UTC m=+1456.526693958" Nov 01 06:34:06.801903 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:34:06.801866 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:34:07.740964 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:07.740932 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-c69zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 06:34:07.741366 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:07.740980 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" podUID="01ed21f8-2d17-47c1-9257-1701c59082f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 06:34:08.744883 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:08.740512 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-c69zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 06:34:08.744883 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:08.740575 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" podUID="01ed21f8-2d17-47c1-9257-1701c59082f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 06:34:09.356991 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:09.356958 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:09.557557 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:09.557532 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:09.739654 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:09.739578 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-c69zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 06:34:09.739901 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:09.739872 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" podUID="01ed21f8-2d17-47c1-9257-1701c59082f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 06:34:10.740105 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:10.740059 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-c69zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 06:34:10.740529 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:10.740122 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" podUID="01ed21f8-2d17-47c1-9257-1701c59082f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 06:34:11.739530 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:11.739499 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-c69zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 06:34:11.739777 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:11.739754 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" podUID="01ed21f8-2d17-47c1-9257-1701c59082f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 06:34:12.740152 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:12.740121 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-c69zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 06:34:12.740550 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:12.740175 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" podUID="01ed21f8-2d17-47c1-9257-1701c59082f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 06:34:13.740078 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:13.740037 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-c69zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 01 06:34:13.740274 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:13.740105 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" podUID="01ed21f8-2d17-47c1-9257-1701c59082f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 01 06:34:14.415889 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:14.415819 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:14.845014 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:14.844957 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:14.845552 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:14.845359 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:14.846394 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:14.846364 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:34:19.420334 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:19.420291 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:21.045023 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:21.044993 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd"] Nov 01 06:34:21.100769 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:21.100735 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9795dbcb-58lbd"] Nov 01 06:34:22.404339 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:22.404268 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2d34f08-dd95-446b-ac13-337294f74d94" path="/var/lib/kubelet/pods/f2d34f08-dd95-446b-ac13-337294f74d94/volumes" Nov 01 06:34:23.422268 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:23.422235 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2"] Nov 01 06:34:23.535525 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:23.535493 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr"] Nov 01 06:34:23.796451 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:23.796361 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:23.947128 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:23.947095 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:24.195896 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:24.184539 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:24.334727 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:24.334694 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:28.423227 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.423184 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2" podUID="d6931d58-6876-4fe4-9f49-f1edba425b49" containerName="payload-processing" containerID="cri-o://141847dda50bd850a0cfc16d1ee6df08e93ca889b63647be442d356ffdff474b" gracePeriod=25 Nov 01 06:34:28.536156 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.536108 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr" podUID="62a2a248-71ad-4331-9b3b-afea7852687d" containerName="payload-pre-processing" containerID="cri-o://66edd178beb0522627137586498727270e453deaa5972c42f5e944b55c6e7094" gracePeriod=25 Nov 01 06:34:28.704489 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.704462 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2" Nov 01 06:34:28.782990 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.782959 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr" Nov 01 06:34:28.794589 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.794557 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a2a248-71ad-4331-9b3b-afea7852687d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a2a248-71ad-4331-9b3b-afea7852687d-plugins-config-volume\") pod \"62a2a248-71ad-4331-9b3b-afea7852687d\" (UID: \"62a2a248-71ad-4331-9b3b-afea7852687d\") " Nov 01 06:34:28.794746 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.794673 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6931d58-6876-4fe4-9f49-f1edba425b49-kube-api-access-kbxbk\" (UniqueName: \"kubernetes.io/projected/d6931d58-6876-4fe4-9f49-f1edba425b49-kube-api-access-kbxbk\") pod \"d6931d58-6876-4fe4-9f49-f1edba425b49\" (UID: \"d6931d58-6876-4fe4-9f49-f1edba425b49\") " Nov 01 06:34:28.794813 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.794769 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6931d58-6876-4fe4-9f49-f1edba425b49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6931d58-6876-4fe4-9f49-f1edba425b49-plugins-config-volume\") pod \"d6931d58-6876-4fe4-9f49-f1edba425b49\" (UID: \"d6931d58-6876-4fe4-9f49-f1edba425b49\") " Nov 01 06:34:28.794813 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.794799 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a2a248-71ad-4331-9b3b-afea7852687d-kube-api-access-gldnw\" (UniqueName: \"kubernetes.io/projected/62a2a248-71ad-4331-9b3b-afea7852687d-kube-api-access-gldnw\") pod \"62a2a248-71ad-4331-9b3b-afea7852687d\" (UID: \"62a2a248-71ad-4331-9b3b-afea7852687d\") " Nov 01 06:34:28.795180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.794991 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a2a248-71ad-4331-9b3b-afea7852687d-plugins-config-volume" pod "62a2a248-71ad-4331-9b3b-afea7852687d" (UID: "62a2a248-71ad-4331-9b3b-afea7852687d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:34:28.795439 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.795412 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62a2a248-71ad-4331-9b3b-afea7852687d-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:28.795439 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.795421 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6931d58-6876-4fe4-9f49-f1edba425b49-plugins-config-volume" pod "d6931d58-6876-4fe4-9f49-f1edba425b49" (UID: "d6931d58-6876-4fe4-9f49-f1edba425b49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:34:28.797656 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.797631 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6931d58-6876-4fe4-9f49-f1edba425b49-kube-api-access-kbxbk" pod "d6931d58-6876-4fe4-9f49-f1edba425b49" (UID: "d6931d58-6876-4fe4-9f49-f1edba425b49"). InnerVolumeSpecName "kube-api-access-kbxbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:34:28.798014 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.797980 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a2a248-71ad-4331-9b3b-afea7852687d-kube-api-access-gldnw" pod "62a2a248-71ad-4331-9b3b-afea7852687d" (UID: "62a2a248-71ad-4331-9b3b-afea7852687d"). InnerVolumeSpecName "kube-api-access-gldnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:34:28.836299 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.836265 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:28.849787 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.849763 4385 generic.go:356] "Generic (PLEG): container finished" podID="d6931d58-6876-4fe4-9f49-f1edba425b49" containerID="141847dda50bd850a0cfc16d1ee6df08e93ca889b63647be442d356ffdff474b" exitCode=0 Nov 01 06:34:28.849976 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.849825 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2" Nov 01 06:34:28.849976 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.849863 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2" event={"ID":"d6931d58-6876-4fe4-9f49-f1edba425b49","Type":"ContainerDied","Data":"141847dda50bd850a0cfc16d1ee6df08e93ca889b63647be442d356ffdff474b"} Nov 01 06:34:28.849976 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.849934 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2" event={"ID":"d6931d58-6876-4fe4-9f49-f1edba425b49","Type":"ContainerDied","Data":"84c73976268aad55edc81e064b3beadec078d9fed0eb50af95fb3f9486e4c353"} Nov 01 06:34:28.849976 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.849965 4385 scope.go:122] "RemoveContainer" containerID="141847dda50bd850a0cfc16d1ee6df08e93ca889b63647be442d356ffdff474b" Nov 01 06:34:28.863050 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.860232 4385 generic.go:356] "Generic (PLEG): container finished" podID="3c777717-c462-4ba3-bdc3-7b423808eb9b" containerID="34276e826555c578a861f2d96c0bc61a9b4851851610ecfa40002414df7d95ce" exitCode=0 Nov 01 06:34:28.863050 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.860373 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" event={"ID":"3c777717-c462-4ba3-bdc3-7b423808eb9b","Type":"ContainerDied","Data":"34276e826555c578a861f2d96c0bc61a9b4851851610ecfa40002414df7d95ce"} Nov 01 06:34:28.863050 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.861396 4385 scope.go:122] "RemoveContainer" containerID="34276e826555c578a861f2d96c0bc61a9b4851851610ecfa40002414df7d95ce" Nov 01 06:34:28.863751 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.863648 4385 scope.go:122] "RemoveContainer" containerID="141847dda50bd850a0cfc16d1ee6df08e93ca889b63647be442d356ffdff474b" Nov 01 06:34:28.869776 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:34:28.869730 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141847dda50bd850a0cfc16d1ee6df08e93ca889b63647be442d356ffdff474b\": container with ID starting with 141847dda50bd850a0cfc16d1ee6df08e93ca889b63647be442d356ffdff474b not found: ID does not exist" containerID="141847dda50bd850a0cfc16d1ee6df08e93ca889b63647be442d356ffdff474b" Nov 01 06:34:28.889985 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.869788 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141847dda50bd850a0cfc16d1ee6df08e93ca889b63647be442d356ffdff474b"} err="failed to get container status \"141847dda50bd850a0cfc16d1ee6df08e93ca889b63647be442d356ffdff474b\": rpc error: code = NotFound desc = could not find container \"141847dda50bd850a0cfc16d1ee6df08e93ca889b63647be442d356ffdff474b\": container with ID starting with 141847dda50bd850a0cfc16d1ee6df08e93ca889b63647be442d356ffdff474b not found: ID does not exist" Nov 01 06:34:28.889985 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.870753 4385 generic.go:356] "Generic (PLEG): container finished" podID="62a2a248-71ad-4331-9b3b-afea7852687d" containerID="66edd178beb0522627137586498727270e453deaa5972c42f5e944b55c6e7094" exitCode=0 Nov 01 06:34:28.889985 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.870790 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr" event={"ID":"62a2a248-71ad-4331-9b3b-afea7852687d","Type":"ContainerDied","Data":"66edd178beb0522627137586498727270e453deaa5972c42f5e944b55c6e7094"} Nov 01 06:34:28.889985 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.870849 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr" event={"ID":"62a2a248-71ad-4331-9b3b-afea7852687d","Type":"ContainerDied","Data":"46a9178f674c116bee9b9d4eb410b7dfced7b4e0146b75e6f2ae2c9fe487f18b"} Nov 01 06:34:28.889985 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.870880 4385 scope.go:122] "RemoveContainer" containerID="66edd178beb0522627137586498727270e453deaa5972c42f5e944b55c6e7094" Nov 01 06:34:28.889985 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.871007 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr" Nov 01 06:34:28.889985 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.888593 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:34:28.889985 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.888721 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:34:28.889985 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.888771 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:34:28.896695 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.896666 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbxbk\" (UniqueName: \"kubernetes.io/projected/d6931d58-6876-4fe4-9f49-f1edba425b49-kube-api-access-kbxbk\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:28.896695 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.896695 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6931d58-6876-4fe4-9f49-f1edba425b49-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:28.896695 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.896705 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gldnw\" (UniqueName: \"kubernetes.io/projected/62a2a248-71ad-4331-9b3b-afea7852687d-kube-api-access-gldnw\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:28.916062 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.916040 4385 scope.go:122] "RemoveContainer" containerID="66edd178beb0522627137586498727270e453deaa5972c42f5e944b55c6e7094" Nov 01 06:34:28.916435 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:34:28.916405 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66edd178beb0522627137586498727270e453deaa5972c42f5e944b55c6e7094\": container with ID starting with 66edd178beb0522627137586498727270e453deaa5972c42f5e944b55c6e7094 not found: ID does not exist" containerID="66edd178beb0522627137586498727270e453deaa5972c42f5e944b55c6e7094" Nov 01 06:34:28.916560 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:28.916448 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66edd178beb0522627137586498727270e453deaa5972c42f5e944b55c6e7094"} err="failed to get container status \"66edd178beb0522627137586498727270e453deaa5972c42f5e944b55c6e7094\": rpc error: code = NotFound desc = could not find container \"66edd178beb0522627137586498727270e453deaa5972c42f5e944b55c6e7094\": container with ID starting with 66edd178beb0522627137586498727270e453deaa5972c42f5e944b55c6e7094 not found: ID does not exist" Nov 01 06:34:29.074726 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:29.074692 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2"] Nov 01 06:34:29.079008 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:29.078987 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-56ae52d5-d44f5c797-mgbm2"] Nov 01 06:34:29.188022 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:29.187939 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr"] Nov 01 06:34:29.191522 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:29.191500 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-56ae52d5-5cf56bddf5-shhdr"] Nov 01 06:34:29.914649 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:29.914619 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" event={"ID":"3c777717-c462-4ba3-bdc3-7b423808eb9b","Type":"ContainerStarted","Data":"40b25a0cfa6ca56cc04ba0b78ecaf6b92f2cbd392f062251712074a67230d8aa"} Nov 01 06:34:30.140204 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:30.140168 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:30.405999 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:30.405948 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a2a248-71ad-4331-9b3b-afea7852687d" path="/var/lib/kubelet/pods/62a2a248-71ad-4331-9b3b-afea7852687d/volumes" Nov 01 06:34:30.406566 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:30.406545 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6931d58-6876-4fe4-9f49-f1edba425b49" path="/var/lib/kubelet/pods/d6931d58-6876-4fe4-9f49-f1edba425b49/volumes" Nov 01 06:34:30.529388 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:30.529356 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:34:30.529659 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:30.529632 4385 patch_prober.go:29] interesting pod/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 06:34:30.529788 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:30.529693 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" podUID="3c777717-c462-4ba3-bdc3-7b423808eb9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 06:34:31.528784 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:31.528704 4385 patch_prober.go:29] interesting pod/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 06:34:31.528784 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:31.528759 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" podUID="3c777717-c462-4ba3-bdc3-7b423808eb9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 06:34:32.529042 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:32.529010 4385 patch_prober.go:29] interesting pod/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 06:34:32.529459 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:32.529070 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" podUID="3c777717-c462-4ba3-bdc3-7b423808eb9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 06:34:33.478861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:33.474445 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:33.530931 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:33.530614 4385 patch_prober.go:29] interesting pod/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 06:34:33.530931 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:33.530686 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" podUID="3c777717-c462-4ba3-bdc3-7b423808eb9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 06:34:34.528817 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:34.528783 4385 patch_prober.go:29] interesting pod/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 06:34:34.529027 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:34.528865 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" podUID="3c777717-c462-4ba3-bdc3-7b423808eb9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 06:34:35.548507 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:35.548482 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:34:35.549775 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:35.549751 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:34:35.550520 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:35.550500 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 06:34:36.784259 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:34:36.784218 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:34:38.182353 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:38.182314 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:42.289886 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:42.289823 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:42.933868 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:42.933793 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:46.123046 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:46.123005 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48"] Nov 01 06:34:46.378730 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:46.378656 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:46.575056 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:46.575020 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:46.731621 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:46.731521 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:47.136861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:47.136799 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:47.899734 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:47.899692 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:50.751693 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.751663 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59"] Nov 01 06:34:50.752441 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.752418 4385 cpu_manager.go:414] "containerMap: removing container" podUID="d6931d58-6876-4fe4-9f49-f1edba425b49" containerName="payload-processing" Nov 01 06:34:50.752555 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.752442 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6931d58-6876-4fe4-9f49-f1edba425b49" containerName="payload-processing" Nov 01 06:34:50.752555 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.752484 4385 cpu_manager.go:414] "containerMap: removing container" podUID="62a2a248-71ad-4331-9b3b-afea7852687d" containerName="payload-pre-processing" Nov 01 06:34:50.752555 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.752492 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a2a248-71ad-4331-9b3b-afea7852687d" containerName="payload-pre-processing" Nov 01 06:34:50.752710 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.752642 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6931d58-6876-4fe4-9f49-f1edba425b49" containerName="payload-processing" Nov 01 06:34:50.752710 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.752661 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a2a248-71ad-4331-9b3b-afea7852687d" containerName="payload-pre-processing" Nov 01 06:34:50.756170 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.756148 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.799823 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.799794 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a15f2ea-5eb8-45db-aa03-667b38836e18-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.799982 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.799846 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qshmg\" (UniqueName: \"kubernetes.io/projected/7a15f2ea-5eb8-45db-aa03-667b38836e18-kube-api-access-qshmg\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.799982 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.799875 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.800125 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.800041 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.800125 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.800087 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.800215 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.800174 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.800323 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.800228 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.800323 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.800293 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.800427 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.800332 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.901795 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.901755 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a15f2ea-5eb8-45db-aa03-667b38836e18-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.902047 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.901804 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qshmg\" (UniqueName: \"kubernetes.io/projected/7a15f2ea-5eb8-45db-aa03-667b38836e18-kube-api-access-qshmg\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.902047 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.901858 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.902047 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.901910 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.902047 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.901941 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.902047 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.901995 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.902349 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.902327 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.902409 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.902376 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.902561 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.902541 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.902758 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.902737 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.902897 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.902777 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.902897 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.902609 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.902897 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.902623 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a15f2ea-5eb8-45db-aa03-667b38836e18-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.903178 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.903127 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.905156 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.905126 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.905651 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.905629 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.928717 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.928692 4385 scope.go:122] "RemoveContainer" containerID="ce332e32bd102f1c79c5687a1645dcf11cb2488eea9897d44024b3c62a13baf7" Nov 01 06:34:50.944456 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.944438 4385 scope.go:122] "RemoveContainer" containerID="35cc4a7e764a28bedb8271d3c8991b0f161bb9a7dfe717b62354c63c6c8b57fd" Nov 01 06:34:50.954986 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.954956 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59"] Nov 01 06:34:50.957941 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.957822 4385 scope.go:122] "RemoveContainer" containerID="b3a8fb4b942112e862fa71834d73a60bfed5d94da7423304e8b534e46bead324" Nov 01 06:34:50.961043 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.961018 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:50.961155 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:50.961111 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qshmg\" (UniqueName: \"kubernetes.io/projected/7a15f2ea-5eb8-45db-aa03-667b38836e18-kube-api-access-qshmg\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2tj59\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:51.069040 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:51.069011 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:51.123785 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:51.123694 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48" podUID="fca2f250-9e02-4618-a254-da9e55299134" containerName="payload-pre-processing" containerID="cri-o://7c526a2cef825c0dab8745736e7b2ec62744e71fc6c88cc7042eef07788d4f17" gracePeriod=25 Nov 01 06:34:51.502791 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:51.502758 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59"] Nov 01 06:34:51.503313 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:34:51.503284 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a15f2ea_5eb8_45db_aa03_667b38836e18.slice/crio-fb7b880f5cddd298f3dd6c693dd89ec533aafbf2a288f39c3110da3c0178b469 WatchSource:0}: Error finding container fb7b880f5cddd298f3dd6c693dd89ec533aafbf2a288f39c3110da3c0178b469: Status 404 returned error can't find the container with id fb7b880f5cddd298f3dd6c693dd89ec533aafbf2a288f39c3110da3c0178b469 Nov 01 06:34:51.506094 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:51.506065 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:34:51.506160 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:51.506128 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:34:51.506203 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:51.506157 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:34:52.098507 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:52.098476 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" event={"ID":"7a15f2ea-5eb8-45db-aa03-667b38836e18","Type":"ContainerStarted","Data":"07113aa204f121e898155e4490ea656b117f6634a02d0f5ffd05077abacf04c6"} Nov 01 06:34:52.098977 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:52.098530 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" event={"ID":"7a15f2ea-5eb8-45db-aa03-667b38836e18","Type":"ContainerStarted","Data":"fb7b880f5cddd298f3dd6c693dd89ec533aafbf2a288f39c3110da3c0178b469"} Nov 01 06:34:52.312718 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:52.312650 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" podStartSLOduration=2.312630708 podStartE2EDuration="2.312630708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:34:52.262115255 +0000 UTC m=+1502.025915113" watchObservedRunningTime="2026-11-01 06:34:52.312630708 +0000 UTC m=+1502.076430565" Nov 01 06:34:52.823966 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:52.823922 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:52.990858 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:52.983288 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:53.069737 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:53.069710 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:34:53.089917 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:53.078098 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2tj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 01 06:34:53.089917 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:53.078169 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" podUID="7a15f2ea-5eb8-45db-aa03-667b38836e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 01 06:34:54.091860 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:54.085986 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2tj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 01 06:34:54.091860 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:54.086053 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" podUID="7a15f2ea-5eb8-45db-aa03-667b38836e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 01 06:34:55.070237 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:55.070200 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2tj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 01 06:34:55.070515 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:55.070485 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" podUID="7a15f2ea-5eb8-45db-aa03-667b38836e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 01 06:34:55.801911 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:55.801657 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48" podUID="fca2f250-9e02-4618-a254-da9e55299134" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:34:56.070951 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:56.069905 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2tj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 01 06:34:56.070951 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:56.069981 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" podUID="7a15f2ea-5eb8-45db-aa03-667b38836e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 01 06:34:57.095865 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:57.095389 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2tj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 01 06:34:57.095865 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:57.095473 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" podUID="7a15f2ea-5eb8-45db-aa03-667b38836e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 01 06:34:57.133863 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:57.124977 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:58.071264 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:58.071228 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2tj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 01 06:34:58.071577 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:58.071548 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" podUID="7a15f2ea-5eb8-45db-aa03-667b38836e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 01 06:34:58.839323 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:58.839283 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:59.036650 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:59.036610 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:59.069808 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:59.069769 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2tj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 01 06:34:59.070064 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:59.069870 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" podUID="7a15f2ea-5eb8-45db-aa03-667b38836e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 01 06:34:59.195998 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:59.195910 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:59.351422 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:59.351387 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:34:59.605184 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:59.605138 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99"] Nov 01 06:34:59.606105 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:59.605738 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" podUID="d92af192-040f-459a-aa4a-bc6e90460eb6" containerName="istio-proxy" containerID="cri-o://99a9032ab0b065d994e134d2e686d17ebfa13a60a0c4744b895bb7aac4b665d7" gracePeriod=30 Nov 01 06:34:59.615544 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:34:59.615500 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:00.211409 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:00.211375 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:00.357439 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:00.357405 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:00.524460 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:00.524367 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:00.726390 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:00.726357 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:35:00.726765 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:00.726736 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:35:00.727767 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:00.727748 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:35:01.448370 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:01.448330 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:01.618428 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:01.618390 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:01.780981 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:01.780895 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:01.934316 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:01.934274 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:02.240433 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:02.240392 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:02.445338 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:02.445306 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:03.037233 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:03.037194 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:03.050518 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:03.050484 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448"] Nov 01 06:35:03.099979 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:03.099933 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d95c9d-xf448"] Nov 01 06:35:03.254733 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:03.252156 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:04.386240 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:04.386204 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dc828d8-18f2-4c17-935c-87572fa0001f" path="/var/lib/kubelet/pods/6dc828d8-18f2-4c17-935c-87572fa0001f/volumes" Nov 01 06:35:04.630738 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:04.630564 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:04.887441 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:04.887413 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:35:04.995922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:04.995239 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-data\") pod \"d92af192-040f-459a-aa4a-bc6e90460eb6\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " Nov 01 06:35:04.995922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:04.995314 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-token\" (UniqueName: \"kubernetes.io/projected/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-token\") pod \"d92af192-040f-459a-aa4a-bc6e90460eb6\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " Nov 01 06:35:04.995922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:04.995382 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-credential-socket\") pod \"d92af192-040f-459a-aa4a-bc6e90460eb6\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " Nov 01 06:35:04.995922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:04.995447 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-podinfo\") pod \"d92af192-040f-459a-aa4a-bc6e90460eb6\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " Nov 01 06:35:04.995922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:04.995504 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-envoy\") pod \"d92af192-040f-459a-aa4a-bc6e90460eb6\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " Nov 01 06:35:04.995922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:04.995538 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d92af192-040f-459a-aa4a-bc6e90460eb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d92af192-040f-459a-aa4a-bc6e90460eb6-istiod-ca-cert\") pod \"d92af192-040f-459a-aa4a-bc6e90460eb6\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " Nov 01 06:35:04.995922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:04.995569 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-workload-socket\") pod \"d92af192-040f-459a-aa4a-bc6e90460eb6\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " Nov 01 06:35:04.995922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:04.995663 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-workload-certs\") pod \"d92af192-040f-459a-aa4a-bc6e90460eb6\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " Nov 01 06:35:04.995922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:04.995697 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d92af192-040f-459a-aa4a-bc6e90460eb6-kube-api-access-x6rdp\" (UniqueName: \"kubernetes.io/projected/d92af192-040f-459a-aa4a-bc6e90460eb6-kube-api-access-x6rdp\") pod \"d92af192-040f-459a-aa4a-bc6e90460eb6\" (UID: \"d92af192-040f-459a-aa4a-bc6e90460eb6\") " Nov 01 06:35:04.998458 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:04.997774 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-data" pod "d92af192-040f-459a-aa4a-bc6e90460eb6" (UID: "d92af192-040f-459a-aa4a-bc6e90460eb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:35:04.998458 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:04.997987 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-credential-socket" pod "d92af192-040f-459a-aa4a-bc6e90460eb6" (UID: "d92af192-040f-459a-aa4a-bc6e90460eb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:35:04.998458 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:04.998238 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d92af192-040f-459a-aa4a-bc6e90460eb6-istiod-ca-cert" pod "d92af192-040f-459a-aa4a-bc6e90460eb6" (UID: "d92af192-040f-459a-aa4a-bc6e90460eb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:35:04.999155 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:04.999123 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-workload-certs" pod "d92af192-040f-459a-aa4a-bc6e90460eb6" (UID: "d92af192-040f-459a-aa4a-bc6e90460eb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:35:04.999479 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:04.999459 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-workload-socket" pod "d92af192-040f-459a-aa4a-bc6e90460eb6" (UID: "d92af192-040f-459a-aa4a-bc6e90460eb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:35:05.017230 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.013853 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d92af192-040f-459a-aa4a-bc6e90460eb6-kube-api-access-x6rdp" pod "d92af192-040f-459a-aa4a-bc6e90460eb6" (UID: "d92af192-040f-459a-aa4a-bc6e90460eb6"). InnerVolumeSpecName "kube-api-access-x6rdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:35:05.017230 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.014411 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-envoy" pod "d92af192-040f-459a-aa4a-bc6e90460eb6" (UID: "d92af192-040f-459a-aa4a-bc6e90460eb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:35:05.017423 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.017394 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-token" pod "d92af192-040f-459a-aa4a-bc6e90460eb6" (UID: "d92af192-040f-459a-aa4a-bc6e90460eb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:35:05.033993 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.033955 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-podinfo" pod "d92af192-040f-459a-aa4a-bc6e90460eb6" (UID: "d92af192-040f-459a-aa4a-bc6e90460eb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:35:05.098851 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.098810 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:05.099036 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.098866 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6rdp\" (UniqueName: \"kubernetes.io/projected/d92af192-040f-459a-aa4a-bc6e90460eb6-kube-api-access-x6rdp\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:05.099036 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.098886 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:05.099036 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.098900 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:05.099036 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.098915 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:05.099036 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.098929 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:05.099036 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.098943 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:05.099036 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.098956 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d92af192-040f-459a-aa4a-bc6e90460eb6-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:05.099036 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.098972 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d92af192-040f-459a-aa4a-bc6e90460eb6-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:05.280675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.280571 4385 generic.go:356] "Generic (PLEG): container finished" podID="d92af192-040f-459a-aa4a-bc6e90460eb6" containerID="99a9032ab0b065d994e134d2e686d17ebfa13a60a0c4744b895bb7aac4b665d7" exitCode=0 Nov 01 06:35:05.280889 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.280798 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" event={"ID":"d92af192-040f-459a-aa4a-bc6e90460eb6","Type":"ContainerDied","Data":"99a9032ab0b065d994e134d2e686d17ebfa13a60a0c4744b895bb7aac4b665d7"} Nov 01 06:35:05.281155 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.280888 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" Nov 01 06:35:05.281155 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.280944 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99" event={"ID":"d92af192-040f-459a-aa4a-bc6e90460eb6","Type":"ContainerDied","Data":"3e126b674b21e1ef0bcf1f9221eb14983b93bd45f97f11206f322a86dcff68f0"} Nov 01 06:35:05.281155 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.281023 4385 scope.go:122] "RemoveContainer" containerID="99a9032ab0b065d994e134d2e686d17ebfa13a60a0c4744b895bb7aac4b665d7" Nov 01 06:35:05.293045 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.293024 4385 scope.go:122] "RemoveContainer" containerID="684c9efd19634e800134e0a14edc61b0c021ec9da0cc76c681febd56589edadf" Nov 01 06:35:05.302875 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.302824 4385 scope.go:122] "RemoveContainer" containerID="99a9032ab0b065d994e134d2e686d17ebfa13a60a0c4744b895bb7aac4b665d7" Nov 01 06:35:05.303196 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:35:05.303176 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a9032ab0b065d994e134d2e686d17ebfa13a60a0c4744b895bb7aac4b665d7\": container with ID starting with 99a9032ab0b065d994e134d2e686d17ebfa13a60a0c4744b895bb7aac4b665d7 not found: ID does not exist" containerID="99a9032ab0b065d994e134d2e686d17ebfa13a60a0c4744b895bb7aac4b665d7" Nov 01 06:35:05.303268 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.303205 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a9032ab0b065d994e134d2e686d17ebfa13a60a0c4744b895bb7aac4b665d7"} err="failed to get container status \"99a9032ab0b065d994e134d2e686d17ebfa13a60a0c4744b895bb7aac4b665d7\": rpc error: code = NotFound desc = could not find container \"99a9032ab0b065d994e134d2e686d17ebfa13a60a0c4744b895bb7aac4b665d7\": container with ID starting with 99a9032ab0b065d994e134d2e686d17ebfa13a60a0c4744b895bb7aac4b665d7 not found: ID does not exist" Nov 01 06:35:05.303268 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.303226 4385 scope.go:122] "RemoveContainer" containerID="684c9efd19634e800134e0a14edc61b0c021ec9da0cc76c681febd56589edadf" Nov 01 06:35:05.303539 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:35:05.303516 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684c9efd19634e800134e0a14edc61b0c021ec9da0cc76c681febd56589edadf\": container with ID starting with 684c9efd19634e800134e0a14edc61b0c021ec9da0cc76c681febd56589edadf not found: ID does not exist" containerID="684c9efd19634e800134e0a14edc61b0c021ec9da0cc76c681febd56589edadf" Nov 01 06:35:05.303601 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.303550 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684c9efd19634e800134e0a14edc61b0c021ec9da0cc76c681febd56589edadf"} err="failed to get container status \"684c9efd19634e800134e0a14edc61b0c021ec9da0cc76c681febd56589edadf\": rpc error: code = NotFound desc = could not find container \"684c9efd19634e800134e0a14edc61b0c021ec9da0cc76c681febd56589edadf\": container with ID starting with 684c9efd19634e800134e0a14edc61b0c021ec9da0cc76c681febd56589edadf not found: ID does not exist" Nov 01 06:35:05.448098 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.448063 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99"] Nov 01 06:35:05.451764 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.451739 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-56ae52d5-openshift-default-796558d487-6gp99"] Nov 01 06:35:05.793410 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.793369 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48" podUID="fca2f250-9e02-4618-a254-da9e55299134" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:35:05.831708 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:05.831673 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:06.033054 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:06.032706 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:06.173896 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:06.173828 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:06.385339 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:06.385305 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d92af192-040f-459a-aa4a-bc6e90460eb6" path="/var/lib/kubelet/pods/d92af192-040f-459a-aa4a-bc6e90460eb6/volumes" Nov 01 06:35:06.779112 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:35:06.779040 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:35:07.047239 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:07.047146 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f5d8a2a-zw9mn"] Nov 01 06:35:07.096171 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:07.096136 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f5d8a2a-zw9mn"] Nov 01 06:35:07.618635 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:07.618601 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:08.393466 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:08.393434 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecb3a967-06bc-4169-b359-64ecf120e4e0" path="/var/lib/kubelet/pods/ecb3a967-06bc-4169-b359-64ecf120e4e0/volumes" Nov 01 06:35:08.727542 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:08.727445 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:08.920030 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:08.920000 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:09.125553 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:09.125523 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx"] Nov 01 06:35:09.126109 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:09.126095 4385 cpu_manager.go:414] "containerMap: removing container" podUID="d92af192-040f-459a-aa4a-bc6e90460eb6" containerName="istio-proxy" Nov 01 06:35:09.126167 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:09.126110 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d92af192-040f-459a-aa4a-bc6e90460eb6" containerName="istio-proxy" Nov 01 06:35:09.126167 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:09.126135 4385 cpu_manager.go:414] "containerMap: removing container" podUID="d92af192-040f-459a-aa4a-bc6e90460eb6" containerName="istio-proxy" Nov 01 06:35:09.126167 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:09.126140 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d92af192-040f-459a-aa4a-bc6e90460eb6" containerName="istio-proxy" Nov 01 06:35:09.126265 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:09.126210 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="d92af192-040f-459a-aa4a-bc6e90460eb6" containerName="istio-proxy" Nov 01 06:35:09.130767 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:09.130747 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx" Nov 01 06:35:09.157477 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:09.157453 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f73cf601-1cce-43de-ae55-71c03ed97c70-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx\" (UID: \"f73cf601-1cce-43de-ae55-71c03ed97c70\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx" Nov 01 06:35:09.157621 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:09.157491 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzvmc\" (UniqueName: \"kubernetes.io/projected/f73cf601-1cce-43de-ae55-71c03ed97c70-kube-api-access-kzvmc\") pod \"payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx\" (UID: \"f73cf601-1cce-43de-ae55-71c03ed97c70\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx" Nov 01 06:35:09.187791 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:09.187758 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx"] Nov 01 06:35:09.253857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:09.229826 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-df0a0131-dockercfg-bqbnx\"" Nov 01 06:35:09.253857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:09.232395 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-df0a0131\"" Nov 01 06:35:09.263965 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:09.263926 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f73cf601-1cce-43de-ae55-71c03ed97c70-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx\" (UID: \"f73cf601-1cce-43de-ae55-71c03ed97c70\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx" Nov 01 06:35:09.264250 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:09.264012 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzvmc\" (UniqueName: \"kubernetes.io/projected/f73cf601-1cce-43de-ae55-71c03ed97c70-kube-api-access-kzvmc\") pod \"payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx\" (UID: \"f73cf601-1cce-43de-ae55-71c03ed97c70\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx" Nov 01 06:35:09.265631 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:09.265601 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f73cf601-1cce-43de-ae55-71c03ed97c70-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx\" (UID: \"f73cf601-1cce-43de-ae55-71c03ed97c70\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx" Nov 01 06:35:09.320447 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:09.320420 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzvmc\" (UniqueName: \"kubernetes.io/projected/f73cf601-1cce-43de-ae55-71c03ed97c70-kube-api-access-kzvmc\") pod \"payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx\" (UID: \"f73cf601-1cce-43de-ae55-71c03ed97c70\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx" Nov 01 06:35:09.444381 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:09.444290 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx" Nov 01 06:35:09.745000 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:09.744914 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:09.827740 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:09.827705 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx"] Nov 01 06:35:10.327117 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:10.327077 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx" event={"ID":"f73cf601-1cce-43de-ae55-71c03ed97c70","Type":"ContainerStarted","Data":"1a4c880c128eff032e27ed49deb8091614a487dc9944b86df3c07763e60c101b"} Nov 01 06:35:10.327321 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:10.327134 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx" Nov 01 06:35:10.327321 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:10.327159 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx" event={"ID":"f73cf601-1cce-43de-ae55-71c03ed97c70","Type":"ContainerStarted","Data":"f1b5c8a29130a0e202714e87646ef60a8ca565982b1828dabe26f080515bea1f"} Nov 01 06:35:10.441824 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:10.441762 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx" podStartSLOduration=1.4417439779999999 podStartE2EDuration="1.441743978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:35:10.439089055 +0000 UTC m=+1520.202888912" watchObservedRunningTime="2026-11-01 06:35:10.441743978 +0000 UTC m=+1520.205543837" Nov 01 06:35:11.200695 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:11.200646 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:11.345260 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:11.345225 4385 generic.go:356] "Generic (PLEG): container finished" podID="fca2f250-9e02-4618-a254-da9e55299134" containerID="7c526a2cef825c0dab8745736e7b2ec62744e71fc6c88cc7042eef07788d4f17" exitCode=0 Nov 01 06:35:11.345529 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:11.345500 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48" event={"ID":"fca2f250-9e02-4618-a254-da9e55299134","Type":"ContainerDied","Data":"7c526a2cef825c0dab8745736e7b2ec62744e71fc6c88cc7042eef07788d4f17"} Nov 01 06:35:11.450706 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:11.450680 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48" Nov 01 06:35:11.491435 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:11.491409 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca2f250-9e02-4618-a254-da9e55299134-kube-api-access-c7kgb\" (UniqueName: \"kubernetes.io/projected/fca2f250-9e02-4618-a254-da9e55299134-kube-api-access-c7kgb\") pod \"fca2f250-9e02-4618-a254-da9e55299134\" (UID: \"fca2f250-9e02-4618-a254-da9e55299134\") " Nov 01 06:35:11.491599 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:11.491546 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fca2f250-9e02-4618-a254-da9e55299134-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca2f250-9e02-4618-a254-da9e55299134-plugins-config-volume\") pod \"fca2f250-9e02-4618-a254-da9e55299134\" (UID: \"fca2f250-9e02-4618-a254-da9e55299134\") " Nov 01 06:35:11.491982 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:11.491957 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca2f250-9e02-4618-a254-da9e55299134-plugins-config-volume" pod "fca2f250-9e02-4618-a254-da9e55299134" (UID: "fca2f250-9e02-4618-a254-da9e55299134"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:35:11.493666 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:11.493646 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca2f250-9e02-4618-a254-da9e55299134-kube-api-access-c7kgb" pod "fca2f250-9e02-4618-a254-da9e55299134" (UID: "fca2f250-9e02-4618-a254-da9e55299134"). InnerVolumeSpecName "kube-api-access-c7kgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:35:11.593223 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:11.593188 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7kgb\" (UniqueName: \"kubernetes.io/projected/fca2f250-9e02-4618-a254-da9e55299134-kube-api-access-c7kgb\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:11.593223 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:11.593220 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca2f250-9e02-4618-a254-da9e55299134-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:12.355062 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:12.355036 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48" Nov 01 06:35:12.355498 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:12.355063 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48" event={"ID":"fca2f250-9e02-4618-a254-da9e55299134","Type":"ContainerDied","Data":"2d688b4e17d191ed2c24d3eaaf7000ae8845148735cc0fdec459945e50852f83"} Nov 01 06:35:12.355498 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:12.355115 4385 scope.go:122] "RemoveContainer" containerID="7c526a2cef825c0dab8745736e7b2ec62744e71fc6c88cc7042eef07788d4f17" Nov 01 06:35:12.522789 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:12.522756 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48"] Nov 01 06:35:12.526478 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:12.526459 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e7b2a3-66d7b4cc6c-m7r48"] Nov 01 06:35:14.124849 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:14.124792 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:14.394909 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:14.394277 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca2f250-9e02-4618-a254-da9e55299134" path="/var/lib/kubelet/pods/fca2f250-9e02-4618-a254-da9e55299134/volumes" Nov 01 06:35:15.916306 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:15.916264 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:16.286646 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:16.286559 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg"] Nov 01 06:35:18.046799 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:18.046762 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e78798ce-fkfwc"] Nov 01 06:35:18.096454 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:18.096420 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e78798ce-fkfwc"] Nov 01 06:35:18.396521 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:18.396480 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa42472-eaf4-4a8a-9b47-64a708c379c5" path="/var/lib/kubelet/pods/3fa42472-eaf4-4a8a-9b47-64a708c379c5/volumes" Nov 01 06:35:19.050636 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:19.050597 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77"] Nov 01 06:35:19.101842 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:19.101784 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b78c0aae-tmq77"] Nov 01 06:35:20.385631 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:20.385584 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f84219e-7934-4975-a98f-1d8b04edbc98" path="/var/lib/kubelet/pods/5f84219e-7934-4975-a98f-1d8b04edbc98/volumes" Nov 01 06:35:20.650741 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:20.650542 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:21.287559 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:21.287521 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" podUID="d8da888f-3fd6-4d71-a5be-ad96ea3fd427" containerName="payload-processing" containerID="cri-o://1a464fdb826c68898673cf159a5369ca2949e59fe09970f7674ac01342c97897" gracePeriod=25 Nov 01 06:35:21.350061 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:21.350034 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx" Nov 01 06:35:21.475157 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:21.472593 4385 generic.go:356] "Generic (PLEG): container finished" podID="d8da888f-3fd6-4d71-a5be-ad96ea3fd427" containerID="1a464fdb826c68898673cf159a5369ca2949e59fe09970f7674ac01342c97897" exitCode=0 Nov 01 06:35:21.475157 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:21.472778 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" event={"ID":"d8da888f-3fd6-4d71-a5be-ad96ea3fd427","Type":"ContainerDied","Data":"1a464fdb826c68898673cf159a5369ca2949e59fe09970f7674ac01342c97897"} Nov 01 06:35:21.582917 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:21.582886 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" Nov 01 06:35:21.760558 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:21.760527 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-routing-overlay-volume\") pod \"d8da888f-3fd6-4d71-a5be-ad96ea3fd427\" (UID: \"d8da888f-3fd6-4d71-a5be-ad96ea3fd427\") " Nov 01 06:35:21.760763 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:21.760635 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-kube-api-access-dw6f5\" (UniqueName: \"kubernetes.io/projected/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-kube-api-access-dw6f5\") pod \"d8da888f-3fd6-4d71-a5be-ad96ea3fd427\" (UID: \"d8da888f-3fd6-4d71-a5be-ad96ea3fd427\") " Nov 01 06:35:21.760763 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:21.760702 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-plugins-config-volume\") pod \"d8da888f-3fd6-4d71-a5be-ad96ea3fd427\" (UID: \"d8da888f-3fd6-4d71-a5be-ad96ea3fd427\") " Nov 01 06:35:21.760763 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:21.760723 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-provider-credentials\") pod \"d8da888f-3fd6-4d71-a5be-ad96ea3fd427\" (UID: \"d8da888f-3fd6-4d71-a5be-ad96ea3fd427\") " Nov 01 06:35:21.760962 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:21.760904 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-routing-overlay-volume" pod "d8da888f-3fd6-4d71-a5be-ad96ea3fd427" (UID: "d8da888f-3fd6-4d71-a5be-ad96ea3fd427"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:35:21.761175 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:21.761149 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-plugins-config-volume" pod "d8da888f-3fd6-4d71-a5be-ad96ea3fd427" (UID: "d8da888f-3fd6-4d71-a5be-ad96ea3fd427"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:35:21.761324 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:21.761304 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:21.761387 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:21.761331 4385 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-routing-overlay-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:21.763125 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:21.763092 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-kube-api-access-dw6f5" pod "d8da888f-3fd6-4d71-a5be-ad96ea3fd427" (UID: "d8da888f-3fd6-4d71-a5be-ad96ea3fd427"). InnerVolumeSpecName "kube-api-access-dw6f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:35:21.763125 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:21.763106 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-provider-credentials" pod "d8da888f-3fd6-4d71-a5be-ad96ea3fd427" (UID: "d8da888f-3fd6-4d71-a5be-ad96ea3fd427"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:35:21.777382 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:21.777338 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:21.863023 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:21.862926 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw6f5\" (UniqueName: \"kubernetes.io/projected/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-kube-api-access-dw6f5\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:21.863023 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:21.863024 4385 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8da888f-3fd6-4d71-a5be-ad96ea3fd427-provider-credentials\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:21.975257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:21.975214 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:22.482978 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:22.482938 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" event={"ID":"d8da888f-3fd6-4d71-a5be-ad96ea3fd427","Type":"ContainerDied","Data":"0f7c5ee6b046669b47fb489ad756f4786743426e9a39d69df8fe66e729855367"} Nov 01 06:35:22.482978 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:22.482966 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg" Nov 01 06:35:22.483487 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:22.483009 4385 scope.go:122] "RemoveContainer" containerID="1a464fdb826c68898673cf159a5369ca2949e59fe09970f7674ac01342c97897" Nov 01 06:35:22.649807 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:22.649727 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg"] Nov 01 06:35:22.696890 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:22.696854 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4dmv6hg"] Nov 01 06:35:23.332439 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:23.332405 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:23.817515 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:23.817422 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:24.385615 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:24.385580 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8da888f-3fd6-4d71-a5be-ad96ea3fd427" path="/var/lib/kubelet/pods/d8da888f-3fd6-4d71-a5be-ad96ea3fd427/volumes" Nov 01 06:35:25.690371 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:25.690334 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:29.245099 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:29.245059 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:30.408556 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:30.408508 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:35.151510 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:35.151471 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:35.297705 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:35.297663 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:35.740950 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:35.740922 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk"] Nov 01 06:35:35.850674 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:35.850641 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc"] Nov 01 06:35:36.116974 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:36.116935 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:36.325423 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:36.325391 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:36.779864 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:35:36.779812 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:35:40.222371 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:40.222332 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:40.742203 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:40.742132 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk" podUID="0bf5005a-16d0-42e0-963d-41ca0553856d" containerName="payload-processing" containerID="cri-o://be8b3967e771d44216a80fe2e2ae69a81cbc19f100511c4857eb07701a07a855" gracePeriod=25 Nov 01 06:35:40.851771 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:40.851731 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc" podUID="75f5ca17-d8a4-4ea0-8803-0ad27a3e42db" containerName="payload-pre-processing" containerID="cri-o://1e05e2535af439ac344f9e24f2da2120e7caa560fa227a5f3237c273ad429883" gracePeriod=25 Nov 01 06:35:40.998599 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:40.998400 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:41.034475 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.034448 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk" Nov 01 06:35:41.081499 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.081467 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bf5005a-16d0-42e0-963d-41ca0553856d-kube-api-access-z7gb8\" (UniqueName: \"kubernetes.io/projected/0bf5005a-16d0-42e0-963d-41ca0553856d-kube-api-access-z7gb8\") pod \"0bf5005a-16d0-42e0-963d-41ca0553856d\" (UID: \"0bf5005a-16d0-42e0-963d-41ca0553856d\") " Nov 01 06:35:41.081674 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.081645 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bf5005a-16d0-42e0-963d-41ca0553856d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf5005a-16d0-42e0-963d-41ca0553856d-plugins-config-volume\") pod \"0bf5005a-16d0-42e0-963d-41ca0553856d\" (UID: \"0bf5005a-16d0-42e0-963d-41ca0553856d\") " Nov 01 06:35:41.082200 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.082174 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf5005a-16d0-42e0-963d-41ca0553856d-plugins-config-volume" pod "0bf5005a-16d0-42e0-963d-41ca0553856d" (UID: "0bf5005a-16d0-42e0-963d-41ca0553856d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:35:41.085065 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.085032 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf5005a-16d0-42e0-963d-41ca0553856d-kube-api-access-z7gb8" pod "0bf5005a-16d0-42e0-963d-41ca0553856d" (UID: "0bf5005a-16d0-42e0-963d-41ca0553856d"). InnerVolumeSpecName "kube-api-access-z7gb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:35:41.147807 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.147781 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc" Nov 01 06:35:41.182964 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.182934 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf5005a-16d0-42e0-963d-41ca0553856d-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:41.182964 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.182960 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7gb8\" (UniqueName: \"kubernetes.io/projected/0bf5005a-16d0-42e0-963d-41ca0553856d-kube-api-access-z7gb8\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:41.284445 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.284361 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f5ca17-d8a4-4ea0-8803-0ad27a3e42db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75f5ca17-d8a4-4ea0-8803-0ad27a3e42db-plugins-config-volume\") pod \"75f5ca17-d8a4-4ea0-8803-0ad27a3e42db\" (UID: \"75f5ca17-d8a4-4ea0-8803-0ad27a3e42db\") " Nov 01 06:35:41.284445 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.284421 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f5ca17-d8a4-4ea0-8803-0ad27a3e42db-kube-api-access-8dcv6\" (UniqueName: \"kubernetes.io/projected/75f5ca17-d8a4-4ea0-8803-0ad27a3e42db-kube-api-access-8dcv6\") pod \"75f5ca17-d8a4-4ea0-8803-0ad27a3e42db\" (UID: \"75f5ca17-d8a4-4ea0-8803-0ad27a3e42db\") " Nov 01 06:35:41.284914 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.284806 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f5ca17-d8a4-4ea0-8803-0ad27a3e42db-plugins-config-volume" pod "75f5ca17-d8a4-4ea0-8803-0ad27a3e42db" (UID: "75f5ca17-d8a4-4ea0-8803-0ad27a3e42db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:35:41.286740 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.286717 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f5ca17-d8a4-4ea0-8803-0ad27a3e42db-kube-api-access-8dcv6" pod "75f5ca17-d8a4-4ea0-8803-0ad27a3e42db" (UID: "75f5ca17-d8a4-4ea0-8803-0ad27a3e42db"). InnerVolumeSpecName "kube-api-access-8dcv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:35:41.385725 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.385684 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75f5ca17-d8a4-4ea0-8803-0ad27a3e42db-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:41.385725 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.385717 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dcv6\" (UniqueName: \"kubernetes.io/projected/75f5ca17-d8a4-4ea0-8803-0ad27a3e42db-kube-api-access-8dcv6\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:41.624301 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.624259 4385 generic.go:356] "Generic (PLEG): container finished" podID="75f5ca17-d8a4-4ea0-8803-0ad27a3e42db" containerID="1e05e2535af439ac344f9e24f2da2120e7caa560fa227a5f3237c273ad429883" exitCode=0 Nov 01 06:35:41.624507 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.624308 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc" Nov 01 06:35:41.624507 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.624357 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc" event={"ID":"75f5ca17-d8a4-4ea0-8803-0ad27a3e42db","Type":"ContainerDied","Data":"1e05e2535af439ac344f9e24f2da2120e7caa560fa227a5f3237c273ad429883"} Nov 01 06:35:41.624507 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.624419 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc" event={"ID":"75f5ca17-d8a4-4ea0-8803-0ad27a3e42db","Type":"ContainerDied","Data":"17b5b32fcf13156e754178adf818887cbdb0ac1a0a93c60e9569a4c9996ca27b"} Nov 01 06:35:41.624507 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.624453 4385 scope.go:122] "RemoveContainer" containerID="1e05e2535af439ac344f9e24f2da2120e7caa560fa227a5f3237c273ad429883" Nov 01 06:35:41.631658 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.626333 4385 generic.go:356] "Generic (PLEG): container finished" podID="0bf5005a-16d0-42e0-963d-41ca0553856d" containerID="be8b3967e771d44216a80fe2e2ae69a81cbc19f100511c4857eb07701a07a855" exitCode=0 Nov 01 06:35:41.631658 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.626396 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk" event={"ID":"0bf5005a-16d0-42e0-963d-41ca0553856d","Type":"ContainerDied","Data":"be8b3967e771d44216a80fe2e2ae69a81cbc19f100511c4857eb07701a07a855"} Nov 01 06:35:41.631658 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.626435 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk" event={"ID":"0bf5005a-16d0-42e0-963d-41ca0553856d","Type":"ContainerDied","Data":"b8497b9386f83097451a05d6340332a717bac3ba777cc01e9627820091be8450"} Nov 01 06:35:41.631658 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.626500 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk" Nov 01 06:35:41.641234 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.641204 4385 scope.go:122] "RemoveContainer" containerID="1e05e2535af439ac344f9e24f2da2120e7caa560fa227a5f3237c273ad429883" Nov 01 06:35:41.648738 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:35:41.648708 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e05e2535af439ac344f9e24f2da2120e7caa560fa227a5f3237c273ad429883\": container with ID starting with 1e05e2535af439ac344f9e24f2da2120e7caa560fa227a5f3237c273ad429883 not found: ID does not exist" containerID="1e05e2535af439ac344f9e24f2da2120e7caa560fa227a5f3237c273ad429883" Nov 01 06:35:41.648905 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.648745 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e05e2535af439ac344f9e24f2da2120e7caa560fa227a5f3237c273ad429883"} err="failed to get container status \"1e05e2535af439ac344f9e24f2da2120e7caa560fa227a5f3237c273ad429883\": rpc error: code = NotFound desc = could not find container \"1e05e2535af439ac344f9e24f2da2120e7caa560fa227a5f3237c273ad429883\": container with ID starting with 1e05e2535af439ac344f9e24f2da2120e7caa560fa227a5f3237c273ad429883 not found: ID does not exist" Nov 01 06:35:41.648905 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.648765 4385 scope.go:122] "RemoveContainer" containerID="be8b3967e771d44216a80fe2e2ae69a81cbc19f100511c4857eb07701a07a855" Nov 01 06:35:41.659096 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.659071 4385 scope.go:122] "RemoveContainer" containerID="be8b3967e771d44216a80fe2e2ae69a81cbc19f100511c4857eb07701a07a855" Nov 01 06:35:41.659440 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:35:41.659417 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be8b3967e771d44216a80fe2e2ae69a81cbc19f100511c4857eb07701a07a855\": container with ID starting with be8b3967e771d44216a80fe2e2ae69a81cbc19f100511c4857eb07701a07a855 not found: ID does not exist" containerID="be8b3967e771d44216a80fe2e2ae69a81cbc19f100511c4857eb07701a07a855" Nov 01 06:35:41.659532 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.659465 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be8b3967e771d44216a80fe2e2ae69a81cbc19f100511c4857eb07701a07a855"} err="failed to get container status \"be8b3967e771d44216a80fe2e2ae69a81cbc19f100511c4857eb07701a07a855\": rpc error: code = NotFound desc = could not find container \"be8b3967e771d44216a80fe2e2ae69a81cbc19f100511c4857eb07701a07a855\": container with ID starting with be8b3967e771d44216a80fe2e2ae69a81cbc19f100511c4857eb07701a07a855 not found: ID does not exist" Nov 01 06:35:41.789639 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.789608 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc"] Nov 01 06:35:41.793938 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.793908 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-390a75-58b567fd8c-pjmpc"] Nov 01 06:35:41.901985 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.901906 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk"] Nov 01 06:35:41.905230 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:41.905206 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-390a75-78f89c5775-l5vtk"] Nov 01 06:35:42.384968 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:42.384936 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bf5005a-16d0-42e0-963d-41ca0553856d" path="/var/lib/kubelet/pods/0bf5005a-16d0-42e0-963d-41ca0553856d/volumes" Nov 01 06:35:42.385375 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:42.385361 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f5ca17-d8a4-4ea0-8803-0ad27a3e42db" path="/var/lib/kubelet/pods/75f5ca17-d8a4-4ea0-8803-0ad27a3e42db/volumes" Nov 01 06:35:42.622345 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:42.622308 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:44.924078 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:44.924044 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:49.276186 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.276141 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp"] Nov 01 06:35:49.276699 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.276685 4385 cpu_manager.go:414] "containerMap: removing container" podUID="fca2f250-9e02-4618-a254-da9e55299134" containerName="payload-pre-processing" Nov 01 06:35:49.276746 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.276701 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca2f250-9e02-4618-a254-da9e55299134" containerName="payload-pre-processing" Nov 01 06:35:49.276746 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.276731 4385 cpu_manager.go:414] "containerMap: removing container" podUID="d8da888f-3fd6-4d71-a5be-ad96ea3fd427" containerName="payload-processing" Nov 01 06:35:49.276746 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.276736 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8da888f-3fd6-4d71-a5be-ad96ea3fd427" containerName="payload-processing" Nov 01 06:35:49.276877 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.276755 4385 cpu_manager.go:414] "containerMap: removing container" podUID="0bf5005a-16d0-42e0-963d-41ca0553856d" containerName="payload-processing" Nov 01 06:35:49.276877 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.276760 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bf5005a-16d0-42e0-963d-41ca0553856d" containerName="payload-processing" Nov 01 06:35:49.276877 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.276772 4385 cpu_manager.go:414] "containerMap: removing container" podUID="75f5ca17-d8a4-4ea0-8803-0ad27a3e42db" containerName="payload-pre-processing" Nov 01 06:35:49.276877 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.276777 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f5ca17-d8a4-4ea0-8803-0ad27a3e42db" containerName="payload-pre-processing" Nov 01 06:35:49.276877 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.276877 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8da888f-3fd6-4d71-a5be-ad96ea3fd427" containerName="payload-processing" Nov 01 06:35:49.277038 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.276892 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bf5005a-16d0-42e0-963d-41ca0553856d" containerName="payload-processing" Nov 01 06:35:49.277038 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.276898 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f5ca17-d8a4-4ea0-8803-0ad27a3e42db" containerName="payload-pre-processing" Nov 01 06:35:49.277038 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.276906 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca2f250-9e02-4618-a254-da9e55299134" containerName="payload-pre-processing" Nov 01 06:35:49.277038 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.276911 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="d92af192-040f-459a-aa4a-bc6e90460eb6" containerName="istio-proxy" Nov 01 06:35:49.281798 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.281771 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" Nov 01 06:35:49.336774 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.336744 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp"] Nov 01 06:35:49.380823 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.380791 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whffg\" (UniqueName: \"kubernetes.io/projected/f9c4f235-61d5-413a-a7ec-63d4ea55af38-kube-api-access-whffg\") pod \"maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp\" (UID: \"f9c4f235-61d5-413a-a7ec-63d4ea55af38\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" Nov 01 06:35:49.381002 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.380938 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9c4f235-61d5-413a-a7ec-63d4ea55af38-maas-api-tls\") pod \"maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp\" (UID: \"f9c4f235-61d5-413a-a7ec-63d4ea55af38\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" Nov 01 06:35:49.381050 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.381003 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c4f235-61d5-413a-a7ec-63d4ea55af38-openshift-service-ca\") pod \"maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp\" (UID: \"f9c4f235-61d5-413a-a7ec-63d4ea55af38\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" Nov 01 06:35:49.381050 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.381033 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c4f235-61d5-413a-a7ec-63d4ea55af38-metrics-certs\") pod \"maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp\" (UID: \"f9c4f235-61d5-413a-a7ec-63d4ea55af38\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" Nov 01 06:35:49.381553 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.381534 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-df0a0131-serving-cert\"" Nov 01 06:35:49.482215 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.482182 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9c4f235-61d5-413a-a7ec-63d4ea55af38-maas-api-tls\") pod \"maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp\" (UID: \"f9c4f235-61d5-413a-a7ec-63d4ea55af38\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" Nov 01 06:35:49.482396 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.482263 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c4f235-61d5-413a-a7ec-63d4ea55af38-openshift-service-ca\") pod \"maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp\" (UID: \"f9c4f235-61d5-413a-a7ec-63d4ea55af38\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" Nov 01 06:35:49.482396 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.482321 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c4f235-61d5-413a-a7ec-63d4ea55af38-metrics-certs\") pod \"maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp\" (UID: \"f9c4f235-61d5-413a-a7ec-63d4ea55af38\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" Nov 01 06:35:49.482396 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:35:49.482346 4385 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-serving-cert: secret "maas-api-e2e-migrate-df0a0131-serving-cert" not found Nov 01 06:35:49.482565 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:35:49.482427 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c4f235-61d5-413a-a7ec-63d4ea55af38-maas-api-tls podName:f9c4f235-61d5-413a-a7ec-63d4ea55af38 nodeName:}" failed. No retries permitted until 2026-11-01 06:35:49.982405707 +0000 UTC m=+1559.746205557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f9c4f235-61d5-413a-a7ec-63d4ea55af38-maas-api-tls") pod "maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" (UID: "f9c4f235-61d5-413a-a7ec-63d4ea55af38") : secret "maas-api-e2e-migrate-df0a0131-serving-cert" not found Nov 01 06:35:49.482565 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.482501 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whffg\" (UniqueName: \"kubernetes.io/projected/f9c4f235-61d5-413a-a7ec-63d4ea55af38-kube-api-access-whffg\") pod \"maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp\" (UID: \"f9c4f235-61d5-413a-a7ec-63d4ea55af38\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" Nov 01 06:35:49.483785 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.483759 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c4f235-61d5-413a-a7ec-63d4ea55af38-openshift-service-ca\") pod \"maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp\" (UID: \"f9c4f235-61d5-413a-a7ec-63d4ea55af38\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" Nov 01 06:35:49.485086 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.485063 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c4f235-61d5-413a-a7ec-63d4ea55af38-metrics-certs\") pod \"maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp\" (UID: \"f9c4f235-61d5-413a-a7ec-63d4ea55af38\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" Nov 01 06:35:49.495524 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.495493 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:49.540717 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.540597 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whffg\" (UniqueName: \"kubernetes.io/projected/f9c4f235-61d5-413a-a7ec-63d4ea55af38-kube-api-access-whffg\") pod \"maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp\" (UID: \"f9c4f235-61d5-413a-a7ec-63d4ea55af38\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" Nov 01 06:35:49.692915 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.692876 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:49.837228 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.837195 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:49.989255 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.989215 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9c4f235-61d5-413a-a7ec-63d4ea55af38-maas-api-tls\") pod \"maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp\" (UID: \"f9c4f235-61d5-413a-a7ec-63d4ea55af38\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" Nov 01 06:35:49.991778 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:49.991753 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9c4f235-61d5-413a-a7ec-63d4ea55af38-maas-api-tls\") pod \"maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp\" (UID: \"f9c4f235-61d5-413a-a7ec-63d4ea55af38\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" Nov 01 06:35:50.195623 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:50.195542 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" Nov 01 06:35:50.574207 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:35:50.574167 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c4f235_61d5_413a_a7ec_63d4ea55af38.slice/crio-fecfde337515ebff396f94c49bd2785c67f6da867197ecc7c6704ed9364764bd WatchSource:0}: Error finding container fecfde337515ebff396f94c49bd2785c67f6da867197ecc7c6704ed9364764bd: Status 404 returned error can't find the container with id fecfde337515ebff396f94c49bd2785c67f6da867197ecc7c6704ed9364764bd Nov 01 06:35:50.574774 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:50.574751 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp"] Nov 01 06:35:50.694151 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:50.694118 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" event={"ID":"f9c4f235-61d5-413a-a7ec-63d4ea55af38","Type":"ContainerStarted","Data":"fecfde337515ebff396f94c49bd2785c67f6da867197ecc7c6704ed9364764bd"} Nov 01 06:35:51.034036 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:51.033998 4385 scope.go:122] "RemoveContainer" containerID="2e12480436b293cef633883dab2424f4c773827e1854cc496c00869228ca2a4c" Nov 01 06:35:51.703399 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:51.703361 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" event={"ID":"f9c4f235-61d5-413a-a7ec-63d4ea55af38","Type":"ContainerStarted","Data":"5f5ab6466bac744fc4260820334a9fb45888cde652e638b7d2213010e154eb66"} Nov 01 06:35:51.703808 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:51.703467 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" Nov 01 06:35:51.816654 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:51.816554 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" podStartSLOduration=1.887502706 podStartE2EDuration="2.816539689s" totalImagesPullingTime="929.036983ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:35:51.81597508 +0000 UTC m=+1561.579774938" watchObservedRunningTime="2026-11-01 06:35:51.816539689 +0000 UTC m=+1561.580339564" Nov 01 06:35:54.105778 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:54.105740 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:54.729135 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:54.729098 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:54.934094 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:54.934063 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:55.610011 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:55.609968 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:35:57.715651 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:57.715623 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" Nov 01 06:35:58.922490 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:35:58.922447 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:03.640459 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:03.640379 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:06.010625 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:06.010585 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:06.198913 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:06.198865 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:06.782534 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:36:06.782498 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:36:08.322377 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:08.322340 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:12.392221 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:12.392181 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:14.117256 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:14.117208 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:18.833954 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:18.833921 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:21.372232 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:21.372190 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:21.560200 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:21.560141 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:21.826749 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:21.826716 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:22.868201 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:22.868159 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:23.350827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:23.350795 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:23.764539 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:23.764459 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:24.139118 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:24.139082 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:24.772861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:24.772564 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:25.122387 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:25.122345 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:27.617427 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:27.617393 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:28.497091 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:28.497061 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:28.729307 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:28.729264 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:28.897314 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:28.897278 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:31.679634 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:31.679598 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:31.875569 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:31.875538 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:33.817725 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:33.817666 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:36.781996 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:36:36.781961 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:36:38.236394 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:38.236361 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:42.962473 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:42.962435 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:43.373463 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:43.373416 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:48.423501 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:48.423458 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:51.088437 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:51.088403 4385 scope.go:122] "RemoveContainer" containerID="626928765bcc8762b092c74a0998a59778f214b113ab1117cb937b730c50aba1" Nov 01 06:36:51.099538 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:51.099512 4385 scope.go:122] "RemoveContainer" containerID="767086cabe1a2a85d028531b86bed3182b4ca867e7c223fcd2b80460b491a92b" Nov 01 06:36:51.109005 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:51.108988 4385 scope.go:122] "RemoveContainer" containerID="2fb15bef1615a710c85a6f65be1da18aef15d7762d45d3d091bce72b9f6735b5" Nov 01 06:36:51.120669 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:51.120647 4385 scope.go:122] "RemoveContainer" containerID="49632d496c0f05d898e1ddc0ef8ea9c9e7f366b073fdf6606c95cec92bb46019" Nov 01 06:36:52.980606 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:52.980566 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:54.340524 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:54.340493 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx"] Nov 01 06:36:54.594301 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:54.594214 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:54.741170 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:54.741136 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:55.911804 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:55.902658 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:57.644800 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:57.644762 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:36:58.047044 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:58.046960 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da5127df-8g48m"] Nov 01 06:36:58.052233 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:58.052207 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da5127df-8g48m"] Nov 01 06:36:58.384802 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:58.384766 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="046657d1-34a3-44d6-88e2-8c269f0c7001" path="/var/lib/kubelet/pods/046657d1-34a3-44d6-88e2-8c269f0c7001/volumes" Nov 01 06:36:59.341858 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:59.341799 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx" podUID="f73cf601-1cce-43de-ae55-71c03ed97c70" containerName="payload-pre-processing" containerID="cri-o://1a4c880c128eff032e27ed49deb8091614a487dc9944b86df3c07763e60c101b" gracePeriod=25 Nov 01 06:36:59.600804 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:59.600742 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx" Nov 01 06:36:59.692543 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:59.692508 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f73cf601-1cce-43de-ae55-71c03ed97c70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f73cf601-1cce-43de-ae55-71c03ed97c70-plugins-config-volume\") pod \"f73cf601-1cce-43de-ae55-71c03ed97c70\" (UID: \"f73cf601-1cce-43de-ae55-71c03ed97c70\") " Nov 01 06:36:59.692760 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:59.692744 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f73cf601-1cce-43de-ae55-71c03ed97c70-kube-api-access-kzvmc\" (UniqueName: \"kubernetes.io/projected/f73cf601-1cce-43de-ae55-71c03ed97c70-kube-api-access-kzvmc\") pod \"f73cf601-1cce-43de-ae55-71c03ed97c70\" (UID: \"f73cf601-1cce-43de-ae55-71c03ed97c70\") " Nov 01 06:36:59.692876 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:59.692851 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73cf601-1cce-43de-ae55-71c03ed97c70-plugins-config-volume" pod "f73cf601-1cce-43de-ae55-71c03ed97c70" (UID: "f73cf601-1cce-43de-ae55-71c03ed97c70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:36:59.693139 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:59.693122 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f73cf601-1cce-43de-ae55-71c03ed97c70-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:59.694925 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:59.694904 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73cf601-1cce-43de-ae55-71c03ed97c70-kube-api-access-kzvmc" pod "f73cf601-1cce-43de-ae55-71c03ed97c70" (UID: "f73cf601-1cce-43de-ae55-71c03ed97c70"). InnerVolumeSpecName "kube-api-access-kzvmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:36:59.794461 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:36:59.794428 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzvmc\" (UniqueName: \"kubernetes.io/projected/f73cf601-1cce-43de-ae55-71c03ed97c70-kube-api-access-kzvmc\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:00.225742 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:00.225703 4385 generic.go:356] "Generic (PLEG): container finished" podID="f73cf601-1cce-43de-ae55-71c03ed97c70" containerID="1a4c880c128eff032e27ed49deb8091614a487dc9944b86df3c07763e60c101b" exitCode=0 Nov 01 06:37:00.225936 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:00.225756 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx" Nov 01 06:37:00.225936 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:00.225783 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx" event={"ID":"f73cf601-1cce-43de-ae55-71c03ed97c70","Type":"ContainerDied","Data":"1a4c880c128eff032e27ed49deb8091614a487dc9944b86df3c07763e60c101b"} Nov 01 06:37:00.225936 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:00.225859 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx" event={"ID":"f73cf601-1cce-43de-ae55-71c03ed97c70","Type":"ContainerDied","Data":"f1b5c8a29130a0e202714e87646ef60a8ca565982b1828dabe26f080515bea1f"} Nov 01 06:37:00.225936 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:00.225885 4385 scope.go:122] "RemoveContainer" containerID="1a4c880c128eff032e27ed49deb8091614a487dc9944b86df3c07763e60c101b" Nov 01 06:37:00.236431 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:00.236412 4385 scope.go:122] "RemoveContainer" containerID="1a4c880c128eff032e27ed49deb8091614a487dc9944b86df3c07763e60c101b" Nov 01 06:37:00.236691 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:37:00.236672 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a4c880c128eff032e27ed49deb8091614a487dc9944b86df3c07763e60c101b\": container with ID starting with 1a4c880c128eff032e27ed49deb8091614a487dc9944b86df3c07763e60c101b not found: ID does not exist" containerID="1a4c880c128eff032e27ed49deb8091614a487dc9944b86df3c07763e60c101b" Nov 01 06:37:00.236739 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:00.236701 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4c880c128eff032e27ed49deb8091614a487dc9944b86df3c07763e60c101b"} err="failed to get container status \"1a4c880c128eff032e27ed49deb8091614a487dc9944b86df3c07763e60c101b\": rpc error: code = NotFound desc = could not find container \"1a4c880c128eff032e27ed49deb8091614a487dc9944b86df3c07763e60c101b\": container with ID starting with 1a4c880c128eff032e27ed49deb8091614a487dc9944b86df3c07763e60c101b not found: ID does not exist" Nov 01 06:37:00.392020 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:00.391987 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx"] Nov 01 06:37:00.440629 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:00.440596 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-df0a0131-77db945fc9-txnsx"] Nov 01 06:37:02.315182 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:02.315147 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:02.386112 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:02.386083 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f73cf601-1cce-43de-ae55-71c03ed97c70" path="/var/lib/kubelet/pods/f73cf601-1cce-43de-ae55-71c03ed97c70/volumes" Nov 01 06:37:06.780323 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:37:06.780287 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:37:07.025127 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:07.025083 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:11.563508 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:11.563473 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:11.718574 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:11.718543 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:11.868227 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:11.868188 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:12.021985 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:12.019490 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:12.164892 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:12.164764 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:16.602118 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:16.602081 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:20.571795 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:20.571752 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp"] Nov 01 06:37:20.572268 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:20.572016 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" podUID="f9c4f235-61d5-413a-a7ec-63d4ea55af38" containerName="maas-api" containerID="cri-o://5f5ab6466bac744fc4260820334a9fb45888cde652e638b7d2213010e154eb66" gracePeriod=30 Nov 01 06:37:20.696350 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:20.696307 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:21.048648 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.048620 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" Nov 01 06:37:21.097316 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.097286 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c4f235-61d5-413a-a7ec-63d4ea55af38-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c4f235-61d5-413a-a7ec-63d4ea55af38-openshift-service-ca\") pod \"f9c4f235-61d5-413a-a7ec-63d4ea55af38\" (UID: \"f9c4f235-61d5-413a-a7ec-63d4ea55af38\") " Nov 01 06:37:21.097518 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.097356 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9c4f235-61d5-413a-a7ec-63d4ea55af38-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c4f235-61d5-413a-a7ec-63d4ea55af38-metrics-certs\") pod \"f9c4f235-61d5-413a-a7ec-63d4ea55af38\" (UID: \"f9c4f235-61d5-413a-a7ec-63d4ea55af38\") " Nov 01 06:37:21.097518 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.097399 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9c4f235-61d5-413a-a7ec-63d4ea55af38-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9c4f235-61d5-413a-a7ec-63d4ea55af38-maas-api-tls\") pod \"f9c4f235-61d5-413a-a7ec-63d4ea55af38\" (UID: \"f9c4f235-61d5-413a-a7ec-63d4ea55af38\") " Nov 01 06:37:21.097810 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.097783 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c4f235-61d5-413a-a7ec-63d4ea55af38-kube-api-access-whffg\" (UniqueName: \"kubernetes.io/projected/f9c4f235-61d5-413a-a7ec-63d4ea55af38-kube-api-access-whffg\") pod \"f9c4f235-61d5-413a-a7ec-63d4ea55af38\" (UID: \"f9c4f235-61d5-413a-a7ec-63d4ea55af38\") " Nov 01 06:37:21.098022 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.097971 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c4f235-61d5-413a-a7ec-63d4ea55af38-openshift-service-ca" pod "f9c4f235-61d5-413a-a7ec-63d4ea55af38" (UID: "f9c4f235-61d5-413a-a7ec-63d4ea55af38"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:37:21.104868 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.100502 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c4f235-61d5-413a-a7ec-63d4ea55af38-kube-api-access-whffg" pod "f9c4f235-61d5-413a-a7ec-63d4ea55af38" (UID: "f9c4f235-61d5-413a-a7ec-63d4ea55af38"). InnerVolumeSpecName "kube-api-access-whffg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:37:21.104868 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.101063 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whffg\" (UniqueName: \"kubernetes.io/projected/f9c4f235-61d5-413a-a7ec-63d4ea55af38-kube-api-access-whffg\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:21.104868 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.101088 4385 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c4f235-61d5-413a-a7ec-63d4ea55af38-openshift-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:21.104868 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.102773 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9c4f235-61d5-413a-a7ec-63d4ea55af38-metrics-certs" pod "f9c4f235-61d5-413a-a7ec-63d4ea55af38" (UID: "f9c4f235-61d5-413a-a7ec-63d4ea55af38"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:37:21.105167 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.105117 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9c4f235-61d5-413a-a7ec-63d4ea55af38-maas-api-tls" pod "f9c4f235-61d5-413a-a7ec-63d4ea55af38" (UID: "f9c4f235-61d5-413a-a7ec-63d4ea55af38"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:37:21.202768 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.202724 4385 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c4f235-61d5-413a-a7ec-63d4ea55af38-metrics-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:21.202768 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.202769 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9c4f235-61d5-413a-a7ec-63d4ea55af38-maas-api-tls\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:21.373620 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.373585 4385 generic.go:356] "Generic (PLEG): container finished" podID="f9c4f235-61d5-413a-a7ec-63d4ea55af38" containerID="5f5ab6466bac744fc4260820334a9fb45888cde652e638b7d2213010e154eb66" exitCode=0 Nov 01 06:37:21.373803 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.373630 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" Nov 01 06:37:21.373803 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.373672 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" event={"ID":"f9c4f235-61d5-413a-a7ec-63d4ea55af38","Type":"ContainerDied","Data":"5f5ab6466bac744fc4260820334a9fb45888cde652e638b7d2213010e154eb66"} Nov 01 06:37:21.373803 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.373744 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp" event={"ID":"f9c4f235-61d5-413a-a7ec-63d4ea55af38","Type":"ContainerDied","Data":"fecfde337515ebff396f94c49bd2785c67f6da867197ecc7c6704ed9364764bd"} Nov 01 06:37:21.373803 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.373768 4385 scope.go:122] "RemoveContainer" containerID="5f5ab6466bac744fc4260820334a9fb45888cde652e638b7d2213010e154eb66" Nov 01 06:37:21.384528 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.384508 4385 scope.go:122] "RemoveContainer" containerID="5f5ab6466bac744fc4260820334a9fb45888cde652e638b7d2213010e154eb66" Nov 01 06:37:21.384803 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:37:21.384785 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f5ab6466bac744fc4260820334a9fb45888cde652e638b7d2213010e154eb66\": container with ID starting with 5f5ab6466bac744fc4260820334a9fb45888cde652e638b7d2213010e154eb66 not found: ID does not exist" containerID="5f5ab6466bac744fc4260820334a9fb45888cde652e638b7d2213010e154eb66" Nov 01 06:37:21.384888 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.384812 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f5ab6466bac744fc4260820334a9fb45888cde652e638b7d2213010e154eb66"} err="failed to get container status \"5f5ab6466bac744fc4260820334a9fb45888cde652e638b7d2213010e154eb66\": rpc error: code = NotFound desc = could not find container \"5f5ab6466bac744fc4260820334a9fb45888cde652e638b7d2213010e154eb66\": container with ID starting with 5f5ab6466bac744fc4260820334a9fb45888cde652e638b7d2213010e154eb66 not found: ID does not exist" Nov 01 06:37:21.536945 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.536913 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp"] Nov 01 06:37:21.540560 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:21.540538 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-df0a0131-cdbf54699-kqwmp"] Nov 01 06:37:22.263801 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:22.263770 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:22.386725 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:22.386685 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c4f235-61d5-413a-a7ec-63d4ea55af38" path="/var/lib/kubelet/pods/f9c4f235-61d5-413a-a7ec-63d4ea55af38/volumes" Nov 01 06:37:22.451594 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:22.451557 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:22.609799 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:22.609763 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:22.825437 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:22.825404 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:23.434033 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:23.433991 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:24.040361 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:24.032691 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:24.523378 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:24.523341 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:27.439289 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:27.439243 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:27.654056 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:27.654019 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:27.817658 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:27.817569 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:30.709708 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:30.709671 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:30.879010 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:30.878975 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:32.928149 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:32.928059 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:36.782491 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:37:36.782441 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:37:37.924555 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:37.924519 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:42.611729 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:42.611685 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:43.300086 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:43.300053 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:47.329165 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:47.329125 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:52.027862 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:52.027778 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:52.301961 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:52.301877 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:52.452653 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:52.452616 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:37:56.828272 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:37:56.828234 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:01.541447 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:01.541402 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:05.317396 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:05.317359 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:05.767893 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:05.767853 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:06.371600 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:06.371567 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:06.583532 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:06.583496 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:06.733486 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:06.733396 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:06.783143 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:38:06.783106 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:38:08.709041 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:08.708994 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:10.331012 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:10.330979 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:10.624632 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:10.624543 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:11.371809 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:11.371776 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:11.540368 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:11.540327 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:11.694418 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:11.694324 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:14.426563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:14.426522 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:14.574185 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:14.574150 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:14.726008 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:14.725910 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:16.754519 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:16.754479 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:21.523315 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:21.523279 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:26.218914 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:26.218810 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:30.864113 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:30.864075 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:35.516484 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:35.516446 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:36.783997 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:38:36.783955 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:38:40.209821 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:40.209776 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:43.630267 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:43.630217 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:44.919884 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:44.919855 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:49.647031 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:49.646996 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:51.188003 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:51.187969 4385 scope.go:122] "RemoveContainer" containerID="1789b5ed4ad213344b902ddb1250ffd650b123359e3942796e0b72a2d23fe1c5" Nov 01 06:38:54.291828 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:54.291786 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:58.870009 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:58.869962 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:59.017778 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:59.017743 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:38:59.159871 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:38:59.159754 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:03.726757 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:03.726657 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:06.780672 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:39:06.780638 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:39:08.421706 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:08.421660 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:13.878300 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:13.878263 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:18.969052 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:18.969010 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:22.643879 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:22.643847 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:22.783413 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:22.783377 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:24.264497 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:24.264460 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:29.075451 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:29.075409 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:33.733041 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:33.733003 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:36.779471 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:39:36.779436 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:39:37.116435 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:37.116396 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:38.473886 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:38.473805 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:38.855904 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:38.855870 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:39.041563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:39.041530 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:43.131239 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:43.131204 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:43.969445 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:43.969410 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:47.826628 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:47.826593 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:49.858346 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:49.858300 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:52.530205 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:52.530173 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:56.618854 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:56.617919 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:58.142802 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:58.142761 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:58.422215 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:58.422137 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:39:58.562967 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:39:58.562927 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:00.315397 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:00.315362 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:00.936654 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:00.936623 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:01.673948 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:01.670812 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:01.936692 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:01.936599 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:02.549882 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:02.549849 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:03.766733 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:03.766694 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:04.734857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:04.734448 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:04.956935 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:04.949377 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:05.113683 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:05.113654 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:06.569570 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:06.569520 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:06.783604 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:40:06.783561 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:40:08.915577 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:08.915543 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:11.037542 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:11.037512 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:11.205206 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:11.205168 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:13.699942 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:13.699910 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:18.378727 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:18.378691 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:23.031596 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:23.031556 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:23.241029 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:23.240996 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs"] Nov 01 06:40:23.241211 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:23.241192 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" podUID="01ed21f8-2d17-47c1-9257-1701c59082f0" containerName="istio-proxy" containerID="cri-o://6b59c3388afdf0cbe6ee101733d58e10a99236a9fe1e38ab0a878a50b7f1da0b" gracePeriod=30 Nov 01 06:40:26.763440 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:26.763403 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:27.799580 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:27.799545 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:28.529531 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.529466 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:40:28.621985 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.621957 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-credential-socket\") pod \"01ed21f8-2d17-47c1-9257-1701c59082f0\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " Nov 01 06:40:28.622155 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.621999 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-token\" (UniqueName: \"kubernetes.io/projected/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-token\") pod \"01ed21f8-2d17-47c1-9257-1701c59082f0\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " Nov 01 06:40:28.622155 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.622035 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-workload-socket\") pod \"01ed21f8-2d17-47c1-9257-1701c59082f0\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " Nov 01 06:40:28.622155 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.622096 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ed21f8-2d17-47c1-9257-1701c59082f0-kube-api-access-r8s44\" (UniqueName: \"kubernetes.io/projected/01ed21f8-2d17-47c1-9257-1701c59082f0-kube-api-access-r8s44\") pod \"01ed21f8-2d17-47c1-9257-1701c59082f0\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " Nov 01 06:40:28.622155 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.622152 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-data\") pod \"01ed21f8-2d17-47c1-9257-1701c59082f0\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " Nov 01 06:40:28.622356 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.622199 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-envoy\") pod \"01ed21f8-2d17-47c1-9257-1701c59082f0\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " Nov 01 06:40:28.622356 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.622275 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-credential-socket" pod "01ed21f8-2d17-47c1-9257-1701c59082f0" (UID: "01ed21f8-2d17-47c1-9257-1701c59082f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:28.622356 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.622308 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-podinfo\") pod \"01ed21f8-2d17-47c1-9257-1701c59082f0\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " Nov 01 06:40:28.622511 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.622353 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-workload-socket" pod "01ed21f8-2d17-47c1-9257-1701c59082f0" (UID: "01ed21f8-2d17-47c1-9257-1701c59082f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:28.622511 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.622371 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01ed21f8-2d17-47c1-9257-1701c59082f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ed21f8-2d17-47c1-9257-1701c59082f0-istiod-ca-cert\") pod \"01ed21f8-2d17-47c1-9257-1701c59082f0\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " Nov 01 06:40:28.622511 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.622426 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-workload-certs\") pod \"01ed21f8-2d17-47c1-9257-1701c59082f0\" (UID: \"01ed21f8-2d17-47c1-9257-1701c59082f0\") " Nov 01 06:40:28.622677 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.622422 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-data" pod "01ed21f8-2d17-47c1-9257-1701c59082f0" (UID: "01ed21f8-2d17-47c1-9257-1701c59082f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:28.622735 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.622638 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-workload-certs" pod "01ed21f8-2d17-47c1-9257-1701c59082f0" (UID: "01ed21f8-2d17-47c1-9257-1701c59082f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:28.622735 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.622681 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ed21f8-2d17-47c1-9257-1701c59082f0-istiod-ca-cert" pod "01ed21f8-2d17-47c1-9257-1701c59082f0" (UID: "01ed21f8-2d17-47c1-9257-1701c59082f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:40:28.623164 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.623142 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:28.623231 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.623171 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01ed21f8-2d17-47c1-9257-1701c59082f0-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:28.623231 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.623186 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:28.623231 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.623201 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:28.623231 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.623216 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:28.624619 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.624596 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ed21f8-2d17-47c1-9257-1701c59082f0-kube-api-access-r8s44" pod "01ed21f8-2d17-47c1-9257-1701c59082f0" (UID: "01ed21f8-2d17-47c1-9257-1701c59082f0"). InnerVolumeSpecName "kube-api-access-r8s44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:28.624757 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.624732 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-podinfo" pod "01ed21f8-2d17-47c1-9257-1701c59082f0" (UID: "01ed21f8-2d17-47c1-9257-1701c59082f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:40:28.625009 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.624989 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-envoy" pod "01ed21f8-2d17-47c1-9257-1701c59082f0" (UID: "01ed21f8-2d17-47c1-9257-1701c59082f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:28.625052 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.625004 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-token" pod "01ed21f8-2d17-47c1-9257-1701c59082f0" (UID: "01ed21f8-2d17-47c1-9257-1701c59082f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:28.724158 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.724127 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:28.724158 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.724154 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:28.724158 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.724163 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01ed21f8-2d17-47c1-9257-1701c59082f0-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:28.724158 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.724174 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8s44\" (UniqueName: \"kubernetes.io/projected/01ed21f8-2d17-47c1-9257-1701c59082f0-kube-api-access-r8s44\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:28.871462 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.871424 4385 generic.go:356] "Generic (PLEG): container finished" podID="01ed21f8-2d17-47c1-9257-1701c59082f0" containerID="6b59c3388afdf0cbe6ee101733d58e10a99236a9fe1e38ab0a878a50b7f1da0b" exitCode=0 Nov 01 06:40:28.872007 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.871480 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" Nov 01 06:40:28.872007 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.871496 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" event={"ID":"01ed21f8-2d17-47c1-9257-1701c59082f0","Type":"ContainerDied","Data":"6b59c3388afdf0cbe6ee101733d58e10a99236a9fe1e38ab0a878a50b7f1da0b"} Nov 01 06:40:28.872007 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.871549 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs" event={"ID":"01ed21f8-2d17-47c1-9257-1701c59082f0","Type":"ContainerDied","Data":"3b11b808f4475347ea5521f748468644ee6991b8f0e1d8af31abcc0123137873"} Nov 01 06:40:28.872007 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.871579 4385 scope.go:122] "RemoveContainer" containerID="6b59c3388afdf0cbe6ee101733d58e10a99236a9fe1e38ab0a878a50b7f1da0b" Nov 01 06:40:28.885596 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.885571 4385 scope.go:122] "RemoveContainer" containerID="6b59c3388afdf0cbe6ee101733d58e10a99236a9fe1e38ab0a878a50b7f1da0b" Nov 01 06:40:28.885923 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:40:28.885902 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b59c3388afdf0cbe6ee101733d58e10a99236a9fe1e38ab0a878a50b7f1da0b\": container with ID starting with 6b59c3388afdf0cbe6ee101733d58e10a99236a9fe1e38ab0a878a50b7f1da0b not found: ID does not exist" containerID="6b59c3388afdf0cbe6ee101733d58e10a99236a9fe1e38ab0a878a50b7f1da0b" Nov 01 06:40:28.886023 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:28.885936 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b59c3388afdf0cbe6ee101733d58e10a99236a9fe1e38ab0a878a50b7f1da0b"} err="failed to get container status \"6b59c3388afdf0cbe6ee101733d58e10a99236a9fe1e38ab0a878a50b7f1da0b\": rpc error: code = NotFound desc = could not find container \"6b59c3388afdf0cbe6ee101733d58e10a99236a9fe1e38ab0a878a50b7f1da0b\": container with ID starting with 6b59c3388afdf0cbe6ee101733d58e10a99236a9fe1e38ab0a878a50b7f1da0b not found: ID does not exist" Nov 01 06:40:29.036134 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:29.036099 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs"] Nov 01 06:40:29.041895 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:29.041872 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-c69zs"] Nov 01 06:40:30.384360 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:30.384328 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ed21f8-2d17-47c1-9257-1701c59082f0" path="/var/lib/kubelet/pods/01ed21f8-2d17-47c1-9257-1701c59082f0/volumes" Nov 01 06:40:30.619466 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:30.619430 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:31.519398 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:31.519323 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c"] Nov 01 06:40:31.519801 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:31.519527 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" podUID="010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" containerName="istio-proxy" containerID="cri-o://e9d8e25ac8cac9c328708418d97f4506cf28eb4e03e6bb999ddfaafa832a4443" gracePeriod=30 Nov 01 06:40:32.300983 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:32.300942 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:32.384621 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:32.384589 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg"] Nov 01 06:40:32.384862 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:32.384809 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" podUID="4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63" containerName="istio-proxy" containerID="cri-o://321811122f3e95c3fe018aac6fa2b26a61fc2487d810a669b1685f900e12eb90" gracePeriod=30 Nov 01 06:40:32.399158 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:32.399133 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5"] Nov 01 06:40:32.399346 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:32.399311 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" podUID="51691383-996f-4d97-b9ab-e88457d392fc" containerName="istio-proxy" containerID="cri-o://917842658bf121eb1530e653bef46b2d0a7ae80dfc057068cd27f5e7bde80501" gracePeriod=30 Nov 01 06:40:32.462604 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:32.462570 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:34.970204 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:34.970175 4385 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:40:34.970612 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:34.970240 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" podUID="010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:40:35.110472 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:35.110438 4385 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:40:35.110634 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:35.110495 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" podUID="4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:40:36.781373 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:40:36.781331 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:40:36.790498 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.790471 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:40:36.925213 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.925180 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-envoy\") pod \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " Nov 01 06:40:36.925387 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.925299 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-workload-socket\") pod \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " Nov 01 06:40:36.925453 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.925382 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istiod-ca-cert\") pod \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " Nov 01 06:40:36.925516 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.925453 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-workload-certs\") pod \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " Nov 01 06:40:36.925574 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.925530 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-credential-socket\") pod \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " Nov 01 06:40:36.925626 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.925559 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-workload-socket" pod "010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" (UID: "010ccc17-ec4d-4ec3-b3c9-0704c9982fd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:36.925626 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.925579 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-podinfo\") pod \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " Nov 01 06:40:36.925725 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.925632 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-token\" (UniqueName: \"kubernetes.io/projected/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-token\") pod \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " Nov 01 06:40:36.925725 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.925659 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-data\") pod \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " Nov 01 06:40:36.925725 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.925699 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-kube-api-access-96mgf\" (UniqueName: \"kubernetes.io/projected/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-kube-api-access-96mgf\") pod \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\" (UID: \"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4\") " Nov 01 06:40:36.925894 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.925823 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-credential-socket" pod "010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" (UID: "010ccc17-ec4d-4ec3-b3c9-0704c9982fd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:36.925957 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.925900 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istiod-ca-cert" pod "010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" (UID: "010ccc17-ec4d-4ec3-b3c9-0704c9982fd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:40:36.926272 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.926238 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-data" pod "010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" (UID: "010ccc17-ec4d-4ec3-b3c9-0704c9982fd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:36.926396 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.926304 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-workload-certs" pod "010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" (UID: "010ccc17-ec4d-4ec3-b3c9-0704c9982fd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:36.926461 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.926393 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:36.926461 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.926431 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:36.926461 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.926448 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:36.926614 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.926463 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:36.928482 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.928448 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-token" pod "010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" (UID: "010ccc17-ec4d-4ec3-b3c9-0704c9982fd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:36.928596 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.928482 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-kube-api-access-96mgf" pod "010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" (UID: "010ccc17-ec4d-4ec3-b3c9-0704c9982fd4"). InnerVolumeSpecName "kube-api-access-96mgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:36.928596 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.928473 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-podinfo" pod "010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" (UID: "010ccc17-ec4d-4ec3-b3c9-0704c9982fd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:40:36.929133 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.929110 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-envoy" pod "010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" (UID: "010ccc17-ec4d-4ec3-b3c9-0704c9982fd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:36.940990 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.940961 4385 generic.go:356] "Generic (PLEG): container finished" podID="010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" containerID="e9d8e25ac8cac9c328708418d97f4506cf28eb4e03e6bb999ddfaafa832a4443" exitCode=0 Nov 01 06:40:36.941101 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.941022 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" Nov 01 06:40:36.941101 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.941054 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" event={"ID":"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4","Type":"ContainerDied","Data":"e9d8e25ac8cac9c328708418d97f4506cf28eb4e03e6bb999ddfaafa832a4443"} Nov 01 06:40:36.941218 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.941120 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c" event={"ID":"010ccc17-ec4d-4ec3-b3c9-0704c9982fd4","Type":"ContainerDied","Data":"05745a1b63954b13851bc9f278764985f5054bae315cada50b6f8933c0ad01fa"} Nov 01 06:40:36.941218 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.941151 4385 scope.go:122] "RemoveContainer" containerID="e9d8e25ac8cac9c328708418d97f4506cf28eb4e03e6bb999ddfaafa832a4443" Nov 01 06:40:36.965989 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.965958 4385 scope.go:122] "RemoveContainer" containerID="e9d8e25ac8cac9c328708418d97f4506cf28eb4e03e6bb999ddfaafa832a4443" Nov 01 06:40:36.971176 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:40:36.971149 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9d8e25ac8cac9c328708418d97f4506cf28eb4e03e6bb999ddfaafa832a4443\": container with ID starting with e9d8e25ac8cac9c328708418d97f4506cf28eb4e03e6bb999ddfaafa832a4443 not found: ID does not exist" containerID="e9d8e25ac8cac9c328708418d97f4506cf28eb4e03e6bb999ddfaafa832a4443" Nov 01 06:40:36.971305 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:36.971184 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9d8e25ac8cac9c328708418d97f4506cf28eb4e03e6bb999ddfaafa832a4443"} err="failed to get container status \"e9d8e25ac8cac9c328708418d97f4506cf28eb4e03e6bb999ddfaafa832a4443\": rpc error: code = NotFound desc = could not find container \"e9d8e25ac8cac9c328708418d97f4506cf28eb4e03e6bb999ddfaafa832a4443\": container with ID starting with e9d8e25ac8cac9c328708418d97f4506cf28eb4e03e6bb999ddfaafa832a4443 not found: ID does not exist" Nov 01 06:40:37.027670 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.027634 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.027670 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.027669 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.027915 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.027685 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.027915 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.027698 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96mgf\" (UniqueName: \"kubernetes.io/projected/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-kube-api-access-96mgf\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.027915 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.027712 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.073418 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.073385 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c"] Nov 01 06:40:37.076708 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.076688 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ll76c"] Nov 01 06:40:37.120733 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.120703 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:37.668685 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.668617 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:40:37.706432 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.706409 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:40:37.739288 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.739260 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-envoy\") pod \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " Nov 01 06:40:37.739431 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.739309 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-kube-api-access-7qhzr\" (UniqueName: \"kubernetes.io/projected/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-kube-api-access-7qhzr\") pod \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " Nov 01 06:40:37.739431 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.739328 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-workload-socket\") pod \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " Nov 01 06:40:37.739431 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.739374 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-credential-socket\") pod \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " Nov 01 06:40:37.739431 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.739396 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istiod-ca-cert\") pod \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " Nov 01 06:40:37.739644 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.739475 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-data\") pod \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " Nov 01 06:40:37.739644 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.739502 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-token\" (UniqueName: \"kubernetes.io/projected/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-token\") pod \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " Nov 01 06:40:37.739644 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.739571 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-podinfo\") pod \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " Nov 01 06:40:37.739644 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.739634 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-workload-certs\") pod \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\" (UID: \"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63\") " Nov 01 06:40:37.739869 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.739671 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-workload-socket" pod "4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63" (UID: "4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:37.739869 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.739692 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-credential-socket" pod "4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63" (UID: "4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:37.739869 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.739787 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-data" pod "4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63" (UID: "4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:37.740112 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.739901 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istiod-ca-cert" pod "4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63" (UID: "4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:40:37.740112 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.740019 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-workload-certs" pod "4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63" (UID: "4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:37.740289 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.740271 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.740349 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.740292 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.740349 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.740305 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.740349 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.740313 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.740349 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.740325 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.741712 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.741690 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-envoy" pod "4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63" (UID: "4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:37.741811 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.741728 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-kube-api-access-7qhzr" pod "4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63" (UID: "4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63"). InnerVolumeSpecName "kube-api-access-7qhzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:37.741811 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.741783 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-podinfo" pod "4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63" (UID: "4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:40:37.742140 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.742123 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-token" pod "4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63" (UID: "4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:37.841010 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.840982 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-credential-socket\") pod \"51691383-996f-4d97-b9ab-e88457d392fc\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " Nov 01 06:40:37.841415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.841035 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51691383-996f-4d97-b9ab-e88457d392fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51691383-996f-4d97-b9ab-e88457d392fc-istiod-ca-cert\") pod \"51691383-996f-4d97-b9ab-e88457d392fc\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " Nov 01 06:40:37.841415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.841062 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51691383-996f-4d97-b9ab-e88457d392fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51691383-996f-4d97-b9ab-e88457d392fc-istio-podinfo\") pod \"51691383-996f-4d97-b9ab-e88457d392fc\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " Nov 01 06:40:37.841415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.841113 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-istio-envoy\") pod \"51691383-996f-4d97-b9ab-e88457d392fc\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " Nov 01 06:40:37.841415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.841152 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51691383-996f-4d97-b9ab-e88457d392fc-istio-token\" (UniqueName: \"kubernetes.io/projected/51691383-996f-4d97-b9ab-e88457d392fc-istio-token\") pod \"51691383-996f-4d97-b9ab-e88457d392fc\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " Nov 01 06:40:37.841415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.841194 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-workload-certs\") pod \"51691383-996f-4d97-b9ab-e88457d392fc\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " Nov 01 06:40:37.841415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.841236 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51691383-996f-4d97-b9ab-e88457d392fc-kube-api-access-nxtds\" (UniqueName: \"kubernetes.io/projected/51691383-996f-4d97-b9ab-e88457d392fc-kube-api-access-nxtds\") pod \"51691383-996f-4d97-b9ab-e88457d392fc\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " Nov 01 06:40:37.841415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.841275 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-istio-data\") pod \"51691383-996f-4d97-b9ab-e88457d392fc\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " Nov 01 06:40:37.841415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.841282 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-credential-socket" pod "51691383-996f-4d97-b9ab-e88457d392fc" (UID: "51691383-996f-4d97-b9ab-e88457d392fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:37.841415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.841323 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-workload-socket\") pod \"51691383-996f-4d97-b9ab-e88457d392fc\" (UID: \"51691383-996f-4d97-b9ab-e88457d392fc\") " Nov 01 06:40:37.841867 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.841497 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-workload-certs" pod "51691383-996f-4d97-b9ab-e88457d392fc" (UID: "51691383-996f-4d97-b9ab-e88457d392fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:37.841867 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.841516 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51691383-996f-4d97-b9ab-e88457d392fc-istiod-ca-cert" pod "51691383-996f-4d97-b9ab-e88457d392fc" (UID: "51691383-996f-4d97-b9ab-e88457d392fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:40:37.841867 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.841681 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-workload-socket" pod "51691383-996f-4d97-b9ab-e88457d392fc" (UID: "51691383-996f-4d97-b9ab-e88457d392fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:37.841867 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.841695 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-istio-data" pod "51691383-996f-4d97-b9ab-e88457d392fc" (UID: "51691383-996f-4d97-b9ab-e88457d392fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:37.842279 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.841917 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.842279 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.841931 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.842279 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.841941 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51691383-996f-4d97-b9ab-e88457d392fc-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.842279 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.841954 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.842279 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.841967 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.842279 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.841980 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.842279 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.841992 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.842279 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.842005 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qhzr\" (UniqueName: \"kubernetes.io/projected/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63-kube-api-access-7qhzr\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.842279 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.842027 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.843758 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.843734 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51691383-996f-4d97-b9ab-e88457d392fc-istio-podinfo" pod "51691383-996f-4d97-b9ab-e88457d392fc" (UID: "51691383-996f-4d97-b9ab-e88457d392fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:40:37.843936 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.843913 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-istio-envoy" pod "51691383-996f-4d97-b9ab-e88457d392fc" (UID: "51691383-996f-4d97-b9ab-e88457d392fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:37.843984 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.843961 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51691383-996f-4d97-b9ab-e88457d392fc-istio-token" pod "51691383-996f-4d97-b9ab-e88457d392fc" (UID: "51691383-996f-4d97-b9ab-e88457d392fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:37.844039 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.844022 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51691383-996f-4d97-b9ab-e88457d392fc-kube-api-access-nxtds" pod "51691383-996f-4d97-b9ab-e88457d392fc" (UID: "51691383-996f-4d97-b9ab-e88457d392fc"). InnerVolumeSpecName "kube-api-access-nxtds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:37.943947 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.943869 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51691383-996f-4d97-b9ab-e88457d392fc-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.943947 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.943900 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51691383-996f-4d97-b9ab-e88457d392fc-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.943947 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.943918 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxtds\" (UniqueName: \"kubernetes.io/projected/51691383-996f-4d97-b9ab-e88457d392fc-kube-api-access-nxtds\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.943947 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.943934 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51691383-996f-4d97-b9ab-e88457d392fc-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.949918 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.949887 4385 generic.go:356] "Generic (PLEG): container finished" podID="4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63" containerID="321811122f3e95c3fe018aac6fa2b26a61fc2487d810a669b1685f900e12eb90" exitCode=0 Nov 01 06:40:37.950052 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.949945 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" Nov 01 06:40:37.950052 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.949974 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" event={"ID":"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63","Type":"ContainerDied","Data":"321811122f3e95c3fe018aac6fa2b26a61fc2487d810a669b1685f900e12eb90"} Nov 01 06:40:37.950052 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.950039 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg" event={"ID":"4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63","Type":"ContainerDied","Data":"547cd809868b11965b12b24588f42712bab7955361e3604f0d68036e40f3adb4"} Nov 01 06:40:37.950219 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.950061 4385 scope.go:122] "RemoveContainer" containerID="321811122f3e95c3fe018aac6fa2b26a61fc2487d810a669b1685f900e12eb90" Nov 01 06:40:37.951429 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.951407 4385 generic.go:356] "Generic (PLEG): container finished" podID="51691383-996f-4d97-b9ab-e88457d392fc" containerID="917842658bf121eb1530e653bef46b2d0a7ae80dfc057068cd27f5e7bde80501" exitCode=0 Nov 01 06:40:37.951533 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.951452 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" Nov 01 06:40:37.951533 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.951490 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" event={"ID":"51691383-996f-4d97-b9ab-e88457d392fc","Type":"ContainerDied","Data":"917842658bf121eb1530e653bef46b2d0a7ae80dfc057068cd27f5e7bde80501"} Nov 01 06:40:37.951648 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.951538 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5" event={"ID":"51691383-996f-4d97-b9ab-e88457d392fc","Type":"ContainerDied","Data":"13d01a421bd99be5224463354366529d5b096df59faf53d66e322efb03355640"} Nov 01 06:40:37.960695 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.960679 4385 scope.go:122] "RemoveContainer" containerID="321811122f3e95c3fe018aac6fa2b26a61fc2487d810a669b1685f900e12eb90" Nov 01 06:40:37.961084 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:40:37.960965 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321811122f3e95c3fe018aac6fa2b26a61fc2487d810a669b1685f900e12eb90\": container with ID starting with 321811122f3e95c3fe018aac6fa2b26a61fc2487d810a669b1685f900e12eb90 not found: ID does not exist" containerID="321811122f3e95c3fe018aac6fa2b26a61fc2487d810a669b1685f900e12eb90" Nov 01 06:40:37.961084 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.960988 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321811122f3e95c3fe018aac6fa2b26a61fc2487d810a669b1685f900e12eb90"} err="failed to get container status \"321811122f3e95c3fe018aac6fa2b26a61fc2487d810a669b1685f900e12eb90\": rpc error: code = NotFound desc = could not find container \"321811122f3e95c3fe018aac6fa2b26a61fc2487d810a669b1685f900e12eb90\": container with ID starting with 321811122f3e95c3fe018aac6fa2b26a61fc2487d810a669b1685f900e12eb90 not found: ID does not exist" Nov 01 06:40:37.961084 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.961006 4385 scope.go:122] "RemoveContainer" containerID="917842658bf121eb1530e653bef46b2d0a7ae80dfc057068cd27f5e7bde80501" Nov 01 06:40:37.969995 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.969975 4385 scope.go:122] "RemoveContainer" containerID="917842658bf121eb1530e653bef46b2d0a7ae80dfc057068cd27f5e7bde80501" Nov 01 06:40:37.970280 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:40:37.970257 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"917842658bf121eb1530e653bef46b2d0a7ae80dfc057068cd27f5e7bde80501\": container with ID starting with 917842658bf121eb1530e653bef46b2d0a7ae80dfc057068cd27f5e7bde80501 not found: ID does not exist" containerID="917842658bf121eb1530e653bef46b2d0a7ae80dfc057068cd27f5e7bde80501" Nov 01 06:40:37.970347 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:37.970291 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"917842658bf121eb1530e653bef46b2d0a7ae80dfc057068cd27f5e7bde80501"} err="failed to get container status \"917842658bf121eb1530e653bef46b2d0a7ae80dfc057068cd27f5e7bde80501\": rpc error: code = NotFound desc = could not find container \"917842658bf121eb1530e653bef46b2d0a7ae80dfc057068cd27f5e7bde80501\": container with ID starting with 917842658bf121eb1530e653bef46b2d0a7ae80dfc057068cd27f5e7bde80501 not found: ID does not exist" Nov 01 06:40:38.113854 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:38.113789 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg"] Nov 01 06:40:38.117733 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:38.117710 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-lfwxg"] Nov 01 06:40:38.226949 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:38.226873 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5"] Nov 01 06:40:38.235153 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:38.235131 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zktj5"] Nov 01 06:40:38.361893 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:38.361860 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59"] Nov 01 06:40:38.362090 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:38.362066 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" podUID="7a15f2ea-5eb8-45db-aa03-667b38836e18" containerName="istio-proxy" containerID="cri-o://07113aa204f121e898155e4490ea656b117f6634a02d0f5ffd05077abacf04c6" gracePeriod=30 Nov 01 06:40:38.387160 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:38.387126 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" path="/var/lib/kubelet/pods/010ccc17-ec4d-4ec3-b3c9-0704c9982fd4/volumes" Nov 01 06:40:38.387663 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:38.387646 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63" path="/var/lib/kubelet/pods/4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63/volumes" Nov 01 06:40:38.388091 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:38.388075 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51691383-996f-4d97-b9ab-e88457d392fc" path="/var/lib/kubelet/pods/51691383-996f-4d97-b9ab-e88457d392fc/volumes" Nov 01 06:40:41.570849 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:41.570812 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb"] Nov 01 06:40:41.571275 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:41.571022 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" podUID="d360093e-42be-4758-9760-9fe61e233e4e" containerName="istio-proxy" containerID="cri-o://9acf0c21bcca3390bef90409cbe9d661b71bf5bdb3d2a226283293eca273c1be" gracePeriod=30 Nov 01 06:40:41.823354 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:41.823251 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:43.633638 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.633609 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:40:43.815816 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.815732 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-envoy\") pod \"7a15f2ea-5eb8-45db-aa03-667b38836e18\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " Nov 01 06:40:43.815999 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.815980 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-data\") pod \"7a15f2ea-5eb8-45db-aa03-667b38836e18\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " Nov 01 06:40:43.816059 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.816035 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-podinfo\") pod \"7a15f2ea-5eb8-45db-aa03-667b38836e18\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " Nov 01 06:40:43.816746 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.816101 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-workload-certs\") pod \"7a15f2ea-5eb8-45db-aa03-667b38836e18\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " Nov 01 06:40:43.816746 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.816151 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-token\" (UniqueName: \"kubernetes.io/projected/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-token\") pod \"7a15f2ea-5eb8-45db-aa03-667b38836e18\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " Nov 01 06:40:43.816746 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.816195 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a15f2ea-5eb8-45db-aa03-667b38836e18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a15f2ea-5eb8-45db-aa03-667b38836e18-istiod-ca-cert\") pod \"7a15f2ea-5eb8-45db-aa03-667b38836e18\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " Nov 01 06:40:43.816746 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.816225 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a15f2ea-5eb8-45db-aa03-667b38836e18-kube-api-access-qshmg\" (UniqueName: \"kubernetes.io/projected/7a15f2ea-5eb8-45db-aa03-667b38836e18-kube-api-access-qshmg\") pod \"7a15f2ea-5eb8-45db-aa03-667b38836e18\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " Nov 01 06:40:43.816746 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.816299 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-workload-socket\") pod \"7a15f2ea-5eb8-45db-aa03-667b38836e18\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " Nov 01 06:40:43.816746 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.816350 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-credential-socket\") pod \"7a15f2ea-5eb8-45db-aa03-667b38836e18\" (UID: \"7a15f2ea-5eb8-45db-aa03-667b38836e18\") " Nov 01 06:40:43.816746 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.816510 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-workload-certs" pod "7a15f2ea-5eb8-45db-aa03-667b38836e18" (UID: "7a15f2ea-5eb8-45db-aa03-667b38836e18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:43.816746 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.816729 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-data" pod "7a15f2ea-5eb8-45db-aa03-667b38836e18" (UID: "7a15f2ea-5eb8-45db-aa03-667b38836e18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:43.817238 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.817217 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-workload-socket" pod "7a15f2ea-5eb8-45db-aa03-667b38836e18" (UID: "7a15f2ea-5eb8-45db-aa03-667b38836e18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:43.817303 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.817053 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-credential-socket" pod "7a15f2ea-5eb8-45db-aa03-667b38836e18" (UID: "7a15f2ea-5eb8-45db-aa03-667b38836e18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:43.817435 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.817410 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a15f2ea-5eb8-45db-aa03-667b38836e18-istiod-ca-cert" pod "7a15f2ea-5eb8-45db-aa03-667b38836e18" (UID: "7a15f2ea-5eb8-45db-aa03-667b38836e18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:40:43.817721 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.817699 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:43.817791 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.817726 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:43.817791 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.817741 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a15f2ea-5eb8-45db-aa03-667b38836e18-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:43.817791 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.817755 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:43.817791 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.817771 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:43.819346 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.819315 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-envoy" pod "7a15f2ea-5eb8-45db-aa03-667b38836e18" (UID: "7a15f2ea-5eb8-45db-aa03-667b38836e18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:43.819822 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.819796 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-podinfo" pod "7a15f2ea-5eb8-45db-aa03-667b38836e18" (UID: "7a15f2ea-5eb8-45db-aa03-667b38836e18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:40:43.820203 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.820173 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-token" pod "7a15f2ea-5eb8-45db-aa03-667b38836e18" (UID: "7a15f2ea-5eb8-45db-aa03-667b38836e18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:43.820203 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.820188 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a15f2ea-5eb8-45db-aa03-667b38836e18-kube-api-access-qshmg" pod "7a15f2ea-5eb8-45db-aa03-667b38836e18" (UID: "7a15f2ea-5eb8-45db-aa03-667b38836e18"). InnerVolumeSpecName "kube-api-access-qshmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:43.919117 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.919084 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:43.919117 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.919114 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:43.919309 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.919127 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qshmg\" (UniqueName: \"kubernetes.io/projected/7a15f2ea-5eb8-45db-aa03-667b38836e18-kube-api-access-qshmg\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:43.919309 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:43.919137 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a15f2ea-5eb8-45db-aa03-667b38836e18-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:44.004529 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:44.004497 4385 generic.go:356] "Generic (PLEG): container finished" podID="7a15f2ea-5eb8-45db-aa03-667b38836e18" containerID="07113aa204f121e898155e4490ea656b117f6634a02d0f5ffd05077abacf04c6" exitCode=0 Nov 01 06:40:44.004699 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:44.004550 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" Nov 01 06:40:44.004699 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:44.004590 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" event={"ID":"7a15f2ea-5eb8-45db-aa03-667b38836e18","Type":"ContainerDied","Data":"07113aa204f121e898155e4490ea656b117f6634a02d0f5ffd05077abacf04c6"} Nov 01 06:40:44.004699 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:44.004659 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59" event={"ID":"7a15f2ea-5eb8-45db-aa03-667b38836e18","Type":"ContainerDied","Data":"fb7b880f5cddd298f3dd6c693dd89ec533aafbf2a288f39c3110da3c0178b469"} Nov 01 06:40:44.004699 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:44.004689 4385 scope.go:122] "RemoveContainer" containerID="07113aa204f121e898155e4490ea656b117f6634a02d0f5ffd05077abacf04c6" Nov 01 06:40:44.016137 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:44.016121 4385 scope.go:122] "RemoveContainer" containerID="07113aa204f121e898155e4490ea656b117f6634a02d0f5ffd05077abacf04c6" Nov 01 06:40:44.016421 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:40:44.016403 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07113aa204f121e898155e4490ea656b117f6634a02d0f5ffd05077abacf04c6\": container with ID starting with 07113aa204f121e898155e4490ea656b117f6634a02d0f5ffd05077abacf04c6 not found: ID does not exist" containerID="07113aa204f121e898155e4490ea656b117f6634a02d0f5ffd05077abacf04c6" Nov 01 06:40:44.016469 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:44.016431 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07113aa204f121e898155e4490ea656b117f6634a02d0f5ffd05077abacf04c6"} err="failed to get container status \"07113aa204f121e898155e4490ea656b117f6634a02d0f5ffd05077abacf04c6\": rpc error: code = NotFound desc = could not find container \"07113aa204f121e898155e4490ea656b117f6634a02d0f5ffd05077abacf04c6\": container with ID starting with 07113aa204f121e898155e4490ea656b117f6634a02d0f5ffd05077abacf04c6 not found: ID does not exist" Nov 01 06:40:44.170763 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:44.170725 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59"] Nov 01 06:40:44.220139 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:44.220107 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2tj59"] Nov 01 06:40:44.308140 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:44.308110 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:44.384610 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:44.384580 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a15f2ea-5eb8-45db-aa03-667b38836e18" path="/var/lib/kubelet/pods/7a15f2ea-5eb8-45db-aa03-667b38836e18/volumes" Nov 01 06:40:45.642401 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:45.642366 4385 patch_prober.go:29] interesting pod/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:40:45.642872 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:45.642428 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" podUID="d360093e-42be-4758-9760-9fe61e233e4e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:40:46.504472 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.504436 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:46.643904 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.643824 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55"] Nov 01 06:40:46.644365 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.644065 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" podUID="21387363-d464-4483-9f27-d9fcc7e72804" containerName="istio-proxy" containerID="cri-o://cae687cef88a6625c5d098f7b61e787eb6691c8f46aa484a5a0329b6464347b3" gracePeriod=30 Nov 01 06:40:46.830603 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.830581 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:40:46.855850 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.855810 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-workload-certs\") pod \"d360093e-42be-4758-9760-9fe61e233e4e\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " Nov 01 06:40:46.855990 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.855869 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-credential-socket\") pod \"d360093e-42be-4758-9760-9fe61e233e4e\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " Nov 01 06:40:46.855990 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.855902 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d360093e-42be-4758-9760-9fe61e233e4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d360093e-42be-4758-9760-9fe61e233e4e-istio-podinfo\") pod \"d360093e-42be-4758-9760-9fe61e233e4e\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " Nov 01 06:40:46.855990 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.855924 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d360093e-42be-4758-9760-9fe61e233e4e-istio-token\" (UniqueName: \"kubernetes.io/projected/d360093e-42be-4758-9760-9fe61e233e4e-istio-token\") pod \"d360093e-42be-4758-9760-9fe61e233e4e\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " Nov 01 06:40:46.855990 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.855943 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-workload-socket\") pod \"d360093e-42be-4758-9760-9fe61e233e4e\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " Nov 01 06:40:46.856185 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.856106 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d360093e-42be-4758-9760-9fe61e233e4e-kube-api-access-wz6j8\" (UniqueName: \"kubernetes.io/projected/d360093e-42be-4758-9760-9fe61e233e4e-kube-api-access-wz6j8\") pod \"d360093e-42be-4758-9760-9fe61e233e4e\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " Nov 01 06:40:46.856185 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.856166 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-istio-envoy\") pod \"d360093e-42be-4758-9760-9fe61e233e4e\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " Nov 01 06:40:46.856282 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.856201 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-istio-data\") pod \"d360093e-42be-4758-9760-9fe61e233e4e\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " Nov 01 06:40:46.856282 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.856105 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-workload-certs" pod "d360093e-42be-4758-9760-9fe61e233e4e" (UID: "d360093e-42be-4758-9760-9fe61e233e4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:46.856282 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.856265 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-workload-socket" pod "d360093e-42be-4758-9760-9fe61e233e4e" (UID: "d360093e-42be-4758-9760-9fe61e233e4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:46.856434 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.856254 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-credential-socket" pod "d360093e-42be-4758-9760-9fe61e233e4e" (UID: "d360093e-42be-4758-9760-9fe61e233e4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:46.856434 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.856307 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d360093e-42be-4758-9760-9fe61e233e4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d360093e-42be-4758-9760-9fe61e233e4e-istiod-ca-cert\") pod \"d360093e-42be-4758-9760-9fe61e233e4e\" (UID: \"d360093e-42be-4758-9760-9fe61e233e4e\") " Nov 01 06:40:46.856529 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.856483 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-istio-data" pod "d360093e-42be-4758-9760-9fe61e233e4e" (UID: "d360093e-42be-4758-9760-9fe61e233e4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:46.856681 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.856654 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d360093e-42be-4758-9760-9fe61e233e4e-istiod-ca-cert" pod "d360093e-42be-4758-9760-9fe61e233e4e" (UID: "d360093e-42be-4758-9760-9fe61e233e4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:40:46.856856 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.856817 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:46.856968 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.856857 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:46.856968 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.856873 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:46.856968 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.856885 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:46.856968 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.856894 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d360093e-42be-4758-9760-9fe61e233e4e-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:46.858982 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.858961 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d360093e-42be-4758-9760-9fe61e233e4e-istio-token" pod "d360093e-42be-4758-9760-9fe61e233e4e" (UID: "d360093e-42be-4758-9760-9fe61e233e4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:46.859068 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.858992 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d360093e-42be-4758-9760-9fe61e233e4e-kube-api-access-wz6j8" pod "d360093e-42be-4758-9760-9fe61e233e4e" (UID: "d360093e-42be-4758-9760-9fe61e233e4e"). InnerVolumeSpecName "kube-api-access-wz6j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:46.859068 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.859055 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-istio-envoy" pod "d360093e-42be-4758-9760-9fe61e233e4e" (UID: "d360093e-42be-4758-9760-9fe61e233e4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:46.859148 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.859063 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d360093e-42be-4758-9760-9fe61e233e4e-istio-podinfo" pod "d360093e-42be-4758-9760-9fe61e233e4e" (UID: "d360093e-42be-4758-9760-9fe61e233e4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:40:46.957706 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.957672 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d360093e-42be-4758-9760-9fe61e233e4e-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:46.957706 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.957706 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d360093e-42be-4758-9760-9fe61e233e4e-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:46.957706 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.957720 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz6j8\" (UniqueName: \"kubernetes.io/projected/d360093e-42be-4758-9760-9fe61e233e4e-kube-api-access-wz6j8\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:46.958004 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:46.957735 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d360093e-42be-4758-9760-9fe61e233e4e-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:47.026299 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:47.026267 4385 generic.go:356] "Generic (PLEG): container finished" podID="d360093e-42be-4758-9760-9fe61e233e4e" containerID="9acf0c21bcca3390bef90409cbe9d661b71bf5bdb3d2a226283293eca273c1be" exitCode=0 Nov 01 06:40:47.026479 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:47.026324 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" Nov 01 06:40:47.026479 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:47.026356 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" event={"ID":"d360093e-42be-4758-9760-9fe61e233e4e","Type":"ContainerDied","Data":"9acf0c21bcca3390bef90409cbe9d661b71bf5bdb3d2a226283293eca273c1be"} Nov 01 06:40:47.026479 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:47.026413 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb" event={"ID":"d360093e-42be-4758-9760-9fe61e233e4e","Type":"ContainerDied","Data":"22e1e23a37944d77226b1c97314689f68d6bcc500598abce93475766ed2273e6"} Nov 01 06:40:47.026479 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:47.026442 4385 scope.go:122] "RemoveContainer" containerID="9acf0c21bcca3390bef90409cbe9d661b71bf5bdb3d2a226283293eca273c1be" Nov 01 06:40:47.038026 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:47.037999 4385 scope.go:122] "RemoveContainer" containerID="9acf0c21bcca3390bef90409cbe9d661b71bf5bdb3d2a226283293eca273c1be" Nov 01 06:40:47.038283 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:40:47.038265 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9acf0c21bcca3390bef90409cbe9d661b71bf5bdb3d2a226283293eca273c1be\": container with ID starting with 9acf0c21bcca3390bef90409cbe9d661b71bf5bdb3d2a226283293eca273c1be not found: ID does not exist" containerID="9acf0c21bcca3390bef90409cbe9d661b71bf5bdb3d2a226283293eca273c1be" Nov 01 06:40:47.038329 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:47.038293 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9acf0c21bcca3390bef90409cbe9d661b71bf5bdb3d2a226283293eca273c1be"} err="failed to get container status \"9acf0c21bcca3390bef90409cbe9d661b71bf5bdb3d2a226283293eca273c1be\": rpc error: code = NotFound desc = could not find container \"9acf0c21bcca3390bef90409cbe9d661b71bf5bdb3d2a226283293eca273c1be\": container with ID starting with 9acf0c21bcca3390bef90409cbe9d661b71bf5bdb3d2a226283293eca273c1be not found: ID does not exist" Nov 01 06:40:47.191243 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:47.191210 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb"] Nov 01 06:40:47.195802 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:47.195774 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc6e48-openshift-default-86d7ccfcc-d9mnb"] Nov 01 06:40:48.384101 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:48.384064 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d360093e-42be-4758-9760-9fe61e233e4e" path="/var/lib/kubelet/pods/d360093e-42be-4758-9760-9fe61e233e4e/volumes" Nov 01 06:40:50.322349 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.322318 4385 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:40:50.322706 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.322379 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" podUID="21387363-d464-4483-9f27-d9fcc7e72804" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:40:50.896794 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.896756 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6"] Nov 01 06:40:50.897462 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897443 4385 cpu_manager.go:414] "containerMap: removing container" podUID="01ed21f8-2d17-47c1-9257-1701c59082f0" containerName="istio-proxy" Nov 01 06:40:50.897519 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897466 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ed21f8-2d17-47c1-9257-1701c59082f0" containerName="istio-proxy" Nov 01 06:40:50.897519 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897484 4385 cpu_manager.go:414] "containerMap: removing container" podUID="7a15f2ea-5eb8-45db-aa03-667b38836e18" containerName="istio-proxy" Nov 01 06:40:50.897519 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897492 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a15f2ea-5eb8-45db-aa03-667b38836e18" containerName="istio-proxy" Nov 01 06:40:50.897519 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897510 4385 cpu_manager.go:414] "containerMap: removing container" podUID="f73cf601-1cce-43de-ae55-71c03ed97c70" containerName="payload-pre-processing" Nov 01 06:40:50.897519 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897518 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f73cf601-1cce-43de-ae55-71c03ed97c70" containerName="payload-pre-processing" Nov 01 06:40:50.897691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897531 4385 cpu_manager.go:414] "containerMap: removing container" podUID="4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63" containerName="istio-proxy" Nov 01 06:40:50.897691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897539 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63" containerName="istio-proxy" Nov 01 06:40:50.897691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897550 4385 cpu_manager.go:414] "containerMap: removing container" podUID="f9c4f235-61d5-413a-a7ec-63d4ea55af38" containerName="maas-api" Nov 01 06:40:50.897691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897558 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c4f235-61d5-413a-a7ec-63d4ea55af38" containerName="maas-api" Nov 01 06:40:50.897691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897582 4385 cpu_manager.go:414] "containerMap: removing container" podUID="010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" containerName="istio-proxy" Nov 01 06:40:50.897691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897589 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" containerName="istio-proxy" Nov 01 06:40:50.897691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897598 4385 cpu_manager.go:414] "containerMap: removing container" podUID="51691383-996f-4d97-b9ab-e88457d392fc" containerName="istio-proxy" Nov 01 06:40:50.897691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897607 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51691383-996f-4d97-b9ab-e88457d392fc" containerName="istio-proxy" Nov 01 06:40:50.897691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897628 4385 cpu_manager.go:414] "containerMap: removing container" podUID="d360093e-42be-4758-9760-9fe61e233e4e" containerName="istio-proxy" Nov 01 06:40:50.897691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897634 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d360093e-42be-4758-9760-9fe61e233e4e" containerName="istio-proxy" Nov 01 06:40:50.897995 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897743 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="010ccc17-ec4d-4ec3-b3c9-0704c9982fd4" containerName="istio-proxy" Nov 01 06:40:50.897995 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897754 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="51691383-996f-4d97-b9ab-e88457d392fc" containerName="istio-proxy" Nov 01 06:40:50.897995 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897764 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="d360093e-42be-4758-9760-9fe61e233e4e" containerName="istio-proxy" Nov 01 06:40:50.897995 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897776 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ed21f8-2d17-47c1-9257-1701c59082f0" containerName="istio-proxy" Nov 01 06:40:50.897995 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897785 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a15f2ea-5eb8-45db-aa03-667b38836e18" containerName="istio-proxy" Nov 01 06:40:50.897995 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897795 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="f73cf601-1cce-43de-ae55-71c03ed97c70" containerName="payload-pre-processing" Nov 01 06:40:50.897995 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897805 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bdf0b3a-718c-49d1-9dd9-bd90b5a44e63" containerName="istio-proxy" Nov 01 06:40:50.897995 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.897814 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c4f235-61d5-413a-a7ec-63d4ea55af38" containerName="maas-api" Nov 01 06:40:50.901615 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.901597 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" Nov 01 06:40:50.955083 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:50.955049 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6"] Nov 01 06:40:51.000796 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:51.000766 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4c66acdd-serving-cert\"" Nov 01 06:40:51.001559 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:51.001530 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b323a1-6851-4cef-b5e8-0b211671eb6c-openshift-service-ca\") pod \"maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6\" (UID: \"40b323a1-6851-4cef-b5e8-0b211671eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" Nov 01 06:40:51.001698 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:51.001600 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwn2m\" (UniqueName: \"kubernetes.io/projected/40b323a1-6851-4cef-b5e8-0b211671eb6c-kube-api-access-cwn2m\") pod \"maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6\" (UID: \"40b323a1-6851-4cef-b5e8-0b211671eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" Nov 01 06:40:51.001765 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:51.001702 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b323a1-6851-4cef-b5e8-0b211671eb6c-maas-api-tls\") pod \"maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6\" (UID: \"40b323a1-6851-4cef-b5e8-0b211671eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" Nov 01 06:40:51.001765 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:51.001753 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b323a1-6851-4cef-b5e8-0b211671eb6c-metrics-certs\") pod \"maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6\" (UID: \"40b323a1-6851-4cef-b5e8-0b211671eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" Nov 01 06:40:51.101322 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:51.101279 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:51.102720 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:51.102694 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b323a1-6851-4cef-b5e8-0b211671eb6c-metrics-certs\") pod \"maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6\" (UID: \"40b323a1-6851-4cef-b5e8-0b211671eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" Nov 01 06:40:51.102876 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:51.102810 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b323a1-6851-4cef-b5e8-0b211671eb6c-openshift-service-ca\") pod \"maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6\" (UID: \"40b323a1-6851-4cef-b5e8-0b211671eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" Nov 01 06:40:51.102946 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:51.102908 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwn2m\" (UniqueName: \"kubernetes.io/projected/40b323a1-6851-4cef-b5e8-0b211671eb6c-kube-api-access-cwn2m\") pod \"maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6\" (UID: \"40b323a1-6851-4cef-b5e8-0b211671eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" Nov 01 06:40:51.103003 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:51.102977 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b323a1-6851-4cef-b5e8-0b211671eb6c-maas-api-tls\") pod \"maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6\" (UID: \"40b323a1-6851-4cef-b5e8-0b211671eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" Nov 01 06:40:51.103157 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:40:51.103121 4385 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-serving-cert: secret "maas-api-e2e-ait-4c66acdd-serving-cert" not found Nov 01 06:40:51.103279 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:40:51.103198 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b323a1-6851-4cef-b5e8-0b211671eb6c-maas-api-tls podName:40b323a1-6851-4cef-b5e8-0b211671eb6c nodeName:}" failed. No retries permitted until 2026-11-01 06:40:51.60318178 +0000 UTC m=+1861.366981615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40b323a1-6851-4cef-b5e8-0b211671eb6c-maas-api-tls") pod "maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" (UID: "40b323a1-6851-4cef-b5e8-0b211671eb6c") : secret "maas-api-e2e-ait-4c66acdd-serving-cert" not found Nov 01 06:40:51.103422 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:51.103404 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b323a1-6851-4cef-b5e8-0b211671eb6c-openshift-service-ca\") pod \"maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6\" (UID: \"40b323a1-6851-4cef-b5e8-0b211671eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" Nov 01 06:40:51.105215 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:51.105190 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b323a1-6851-4cef-b5e8-0b211671eb6c-metrics-certs\") pod \"maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6\" (UID: \"40b323a1-6851-4cef-b5e8-0b211671eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" Nov 01 06:40:51.160298 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:51.160231 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwn2m\" (UniqueName: \"kubernetes.io/projected/40b323a1-6851-4cef-b5e8-0b211671eb6c-kube-api-access-cwn2m\") pod \"maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6\" (UID: \"40b323a1-6851-4cef-b5e8-0b211671eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" Nov 01 06:40:51.304669 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:51.304573 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:51.452333 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:51.452255 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:51.609227 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:51.609198 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b323a1-6851-4cef-b5e8-0b211671eb6c-maas-api-tls\") pod \"maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6\" (UID: \"40b323a1-6851-4cef-b5e8-0b211671eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" Nov 01 06:40:51.611858 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:51.611815 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b323a1-6851-4cef-b5e8-0b211671eb6c-maas-api-tls\") pod \"maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6\" (UID: \"40b323a1-6851-4cef-b5e8-0b211671eb6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" Nov 01 06:40:51.816775 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:51.816736 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" Nov 01 06:40:51.925089 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:51.925053 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:40:52.013982 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.013891 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21387363-d464-4483-9f27-d9fcc7e72804-istio-token\" (UniqueName: \"kubernetes.io/projected/21387363-d464-4483-9f27-d9fcc7e72804-istio-token\") pod \"21387363-d464-4483-9f27-d9fcc7e72804\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " Nov 01 06:40:52.013982 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.013970 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-credential-socket\") pod \"21387363-d464-4483-9f27-d9fcc7e72804\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " Nov 01 06:40:52.014216 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.014005 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21387363-d464-4483-9f27-d9fcc7e72804-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21387363-d464-4483-9f27-d9fcc7e72804-istio-podinfo\") pod \"21387363-d464-4483-9f27-d9fcc7e72804\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " Nov 01 06:40:52.014216 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.014089 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21387363-d464-4483-9f27-d9fcc7e72804-kube-api-access-mdh62\" (UniqueName: \"kubernetes.io/projected/21387363-d464-4483-9f27-d9fcc7e72804-kube-api-access-mdh62\") pod \"21387363-d464-4483-9f27-d9fcc7e72804\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " Nov 01 06:40:52.014216 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.014123 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21387363-d464-4483-9f27-d9fcc7e72804-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21387363-d464-4483-9f27-d9fcc7e72804-istiod-ca-cert\") pod \"21387363-d464-4483-9f27-d9fcc7e72804\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " Nov 01 06:40:52.014216 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.014186 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-workload-certs\") pod \"21387363-d464-4483-9f27-d9fcc7e72804\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " Nov 01 06:40:52.014429 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.014307 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-credential-socket" pod "21387363-d464-4483-9f27-d9fcc7e72804" (UID: "21387363-d464-4483-9f27-d9fcc7e72804"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:52.014429 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.014364 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-istio-envoy\") pod \"21387363-d464-4483-9f27-d9fcc7e72804\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " Nov 01 06:40:52.014544 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.014447 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-workload-socket\") pod \"21387363-d464-4483-9f27-d9fcc7e72804\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " Nov 01 06:40:52.014544 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.014489 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-istio-data\") pod \"21387363-d464-4483-9f27-d9fcc7e72804\" (UID: \"21387363-d464-4483-9f27-d9fcc7e72804\") " Nov 01 06:40:52.014544 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.014497 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-workload-certs" pod "21387363-d464-4483-9f27-d9fcc7e72804" (UID: "21387363-d464-4483-9f27-d9fcc7e72804"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:52.014690 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.014597 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21387363-d464-4483-9f27-d9fcc7e72804-istiod-ca-cert" pod "21387363-d464-4483-9f27-d9fcc7e72804" (UID: "21387363-d464-4483-9f27-d9fcc7e72804"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:40:52.014746 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.014716 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-workload-socket" pod "21387363-d464-4483-9f27-d9fcc7e72804" (UID: "21387363-d464-4483-9f27-d9fcc7e72804"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:52.014958 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.014936 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:52.015040 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.014959 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:52.015040 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.014974 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21387363-d464-4483-9f27-d9fcc7e72804-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:52.015040 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.014987 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:52.015279 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.015088 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-istio-data" pod "21387363-d464-4483-9f27-d9fcc7e72804" (UID: "21387363-d464-4483-9f27-d9fcc7e72804"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:52.016448 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.016423 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21387363-d464-4483-9f27-d9fcc7e72804-istio-podinfo" pod "21387363-d464-4483-9f27-d9fcc7e72804" (UID: "21387363-d464-4483-9f27-d9fcc7e72804"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:40:52.016755 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.016739 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-istio-envoy" pod "21387363-d464-4483-9f27-d9fcc7e72804" (UID: "21387363-d464-4483-9f27-d9fcc7e72804"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:52.016888 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.016826 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21387363-d464-4483-9f27-d9fcc7e72804-istio-token" pod "21387363-d464-4483-9f27-d9fcc7e72804" (UID: "21387363-d464-4483-9f27-d9fcc7e72804"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:52.016995 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.016980 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21387363-d464-4483-9f27-d9fcc7e72804-kube-api-access-mdh62" pod "21387363-d464-4483-9f27-d9fcc7e72804" (UID: "21387363-d464-4483-9f27-d9fcc7e72804"). InnerVolumeSpecName "kube-api-access-mdh62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:52.062654 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.062625 4385 generic.go:356] "Generic (PLEG): container finished" podID="21387363-d464-4483-9f27-d9fcc7e72804" containerID="cae687cef88a6625c5d098f7b61e787eb6691c8f46aa484a5a0329b6464347b3" exitCode=0 Nov 01 06:40:52.062818 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.062683 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" Nov 01 06:40:52.062818 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.062709 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" event={"ID":"21387363-d464-4483-9f27-d9fcc7e72804","Type":"ContainerDied","Data":"cae687cef88a6625c5d098f7b61e787eb6691c8f46aa484a5a0329b6464347b3"} Nov 01 06:40:52.062818 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.062790 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55" event={"ID":"21387363-d464-4483-9f27-d9fcc7e72804","Type":"ContainerDied","Data":"19e3656e30c943b66d1b512c3d60440aa5d9c641185806392811a4f4617a8784"} Nov 01 06:40:52.062989 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.062821 4385 scope.go:122] "RemoveContainer" containerID="cae687cef88a6625c5d098f7b61e787eb6691c8f46aa484a5a0329b6464347b3" Nov 01 06:40:52.075779 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.075757 4385 scope.go:122] "RemoveContainer" containerID="cae687cef88a6625c5d098f7b61e787eb6691c8f46aa484a5a0329b6464347b3" Nov 01 06:40:52.076138 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:40:52.076109 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cae687cef88a6625c5d098f7b61e787eb6691c8f46aa484a5a0329b6464347b3\": container with ID starting with cae687cef88a6625c5d098f7b61e787eb6691c8f46aa484a5a0329b6464347b3 not found: ID does not exist" containerID="cae687cef88a6625c5d098f7b61e787eb6691c8f46aa484a5a0329b6464347b3" Nov 01 06:40:52.076247 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.076149 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae687cef88a6625c5d098f7b61e787eb6691c8f46aa484a5a0329b6464347b3"} err="failed to get container status \"cae687cef88a6625c5d098f7b61e787eb6691c8f46aa484a5a0329b6464347b3\": rpc error: code = NotFound desc = could not find container \"cae687cef88a6625c5d098f7b61e787eb6691c8f46aa484a5a0329b6464347b3\": container with ID starting with cae687cef88a6625c5d098f7b61e787eb6691c8f46aa484a5a0329b6464347b3 not found: ID does not exist" Nov 01 06:40:52.116189 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.116159 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21387363-d464-4483-9f27-d9fcc7e72804-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:52.116319 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.116192 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdh62\" (UniqueName: \"kubernetes.io/projected/21387363-d464-4483-9f27-d9fcc7e72804-kube-api-access-mdh62\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:52.116319 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.116207 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:52.116319 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.116220 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21387363-d464-4483-9f27-d9fcc7e72804-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:52.116319 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.116236 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21387363-d464-4483-9f27-d9fcc7e72804-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:52.208815 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:40:52.208783 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b323a1_6851_4cef_b5e8_0b211671eb6c.slice/crio-1d41ebcfb0dc638ff40441d479d4c97eb408eb9dc977d3286ac6e19b3551b8eb WatchSource:0}: Error finding container 1d41ebcfb0dc638ff40441d479d4c97eb408eb9dc977d3286ac6e19b3551b8eb: Status 404 returned error can't find the container with id 1d41ebcfb0dc638ff40441d479d4c97eb408eb9dc977d3286ac6e19b3551b8eb Nov 01 06:40:52.208972 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.208815 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6"] Nov 01 06:40:52.210355 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.210335 4385 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:40:52.227693 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.227673 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55"] Nov 01 06:40:52.232406 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.232388 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-9kc55"] Nov 01 06:40:52.384529 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:52.384500 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21387363-d464-4483-9f27-d9fcc7e72804" path="/var/lib/kubelet/pods/21387363-d464-4483-9f27-d9fcc7e72804/volumes" Nov 01 06:40:53.076041 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:53.076009 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" event={"ID":"40b323a1-6851-4cef-b5e8-0b211671eb6c","Type":"ContainerStarted","Data":"1d41ebcfb0dc638ff40441d479d4c97eb408eb9dc977d3286ac6e19b3551b8eb"} Nov 01 06:40:54.085941 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:54.085907 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" event={"ID":"40b323a1-6851-4cef-b5e8-0b211671eb6c","Type":"ContainerStarted","Data":"83ed80c9df46dc5e28cf84959a96f61bbd41137bd87880373842aeedbf85d053"} Nov 01 06:40:54.086360 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:54.086058 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" Nov 01 06:40:54.202656 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:54.202601 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" podStartSLOduration=3.342417328 podStartE2EDuration="4.202586575s" totalImagesPullingTime="860.169247ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:40:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:40:54.199794254 +0000 UTC m=+1863.963594113" watchObservedRunningTime="2026-11-01 06:40:54.202586575 +0000 UTC m=+1863.966386454" Nov 01 06:40:55.977552 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:55.977509 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:40:59.935884 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:40:59.935844 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:00.100491 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:00.100460 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" Nov 01 06:41:00.668887 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:00.668851 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:05.337784 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:05.337750 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:06.781819 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:41:06.781777 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:41:10.035611 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:10.035572 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:14.674473 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:14.674421 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:19.311772 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:19.311731 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:24.003156 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:24.003122 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:24.747824 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:24.747788 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:24.894086 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:24.894054 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:28.717597 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:28.717555 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:33.492633 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:33.492580 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:36.622285 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:36.622247 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:36.797813 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:41:36.797773 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:41:36.812514 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:36.812466 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:38.139030 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:38.138996 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:42.928463 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:42.928429 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:44.654818 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:44.654786 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:47.745198 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:47.745160 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:47.945352 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:47.945317 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:51.049456 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.049422 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk"] Nov 01 06:41:51.050143 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.050123 4385 cpu_manager.go:414] "containerMap: removing container" podUID="21387363-d464-4483-9f27-d9fcc7e72804" containerName="istio-proxy" Nov 01 06:41:51.050233 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.050145 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21387363-d464-4483-9f27-d9fcc7e72804" containerName="istio-proxy" Nov 01 06:41:51.050294 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.050285 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="21387363-d464-4483-9f27-d9fcc7e72804" containerName="istio-proxy" Nov 01 06:41:51.053514 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.053495 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" Nov 01 06:41:51.106586 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.106557 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"routing-overlay\"" Nov 01 06:41:51.156383 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.156339 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"kube-root-ca.crt\"" Nov 01 06:41:51.156582 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.156391 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"openshift-service-ca.crt\"" Nov 01 06:41:51.156582 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.156399 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"agc-cred-f6f2772a70c5\"" Nov 01 06:41:51.156582 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.156442 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"agc-cred-c7ba2136e663\"" Nov 01 06:41:51.156582 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.156454 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"agc-cred-d41a0e39961b\"" Nov 01 06:41:51.156582 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.156546 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"agc-cred-294b20a2cf15\"" Nov 01 06:41:51.156582 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.156577 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"payload-processing-external-model-plugins-e2e-extmod-a-25719d\"" Nov 01 06:41:51.156827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.156601 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"payload-processing-external-model-e2e--bf6d8971-dockercfg-6qmzh\"" Nov 01 06:41:51.178086 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.178062 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46bcb2e8-bdb2-4632-911d-d0668846115a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk\" (UID: \"46bcb2e8-bdb2-4632-911d-d0668846115a\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" Nov 01 06:41:51.178200 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.178133 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk\" (UID: \"46bcb2e8-bdb2-4632-911d-d0668846115a\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" Nov 01 06:41:51.178260 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.178244 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p72dx\" (UniqueName: \"kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-kube-api-access-p72dx\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk\" (UID: \"46bcb2e8-bdb2-4632-911d-d0668846115a\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" Nov 01 06:41:51.178308 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.178296 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46bcb2e8-bdb2-4632-911d-d0668846115a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk\" (UID: \"46bcb2e8-bdb2-4632-911d-d0668846115a\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" Nov 01 06:41:51.279651 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.279626 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46bcb2e8-bdb2-4632-911d-d0668846115a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk\" (UID: \"46bcb2e8-bdb2-4632-911d-d0668846115a\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" Nov 01 06:41:51.279765 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.279712 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46bcb2e8-bdb2-4632-911d-d0668846115a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk\" (UID: \"46bcb2e8-bdb2-4632-911d-d0668846115a\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" Nov 01 06:41:51.279765 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.279753 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk\" (UID: \"46bcb2e8-bdb2-4632-911d-d0668846115a\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" Nov 01 06:41:51.279894 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.279778 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p72dx\" (UniqueName: \"kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-kube-api-access-p72dx\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk\" (UID: \"46bcb2e8-bdb2-4632-911d-d0668846115a\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" Nov 01 06:41:51.280546 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.280442 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46bcb2e8-bdb2-4632-911d-d0668846115a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk\" (UID: \"46bcb2e8-bdb2-4632-911d-d0668846115a\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" Nov 01 06:41:51.280546 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.280514 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46bcb2e8-bdb2-4632-911d-d0668846115a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk\" (UID: \"46bcb2e8-bdb2-4632-911d-d0668846115a\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" Nov 01 06:41:51.282717 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.282692 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk\" (UID: \"46bcb2e8-bdb2-4632-911d-d0668846115a\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" Nov 01 06:41:51.336935 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.336887 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p72dx\" (UniqueName: \"kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-kube-api-access-p72dx\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk\" (UID: \"46bcb2e8-bdb2-4632-911d-d0668846115a\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" Nov 01 06:41:51.366810 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:51.366771 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" Nov 01 06:41:52.757266 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:52.757235 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:53.544883 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.544852 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc"] Nov 01 06:41:53.549742 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.549724 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" Nov 01 06:41:53.602568 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.602542 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1ff1dc\"" Nov 01 06:41:53.604078 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.604057 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc\" (UID: \"e5a2a7f9-63f8-44ae-a226-c5fd9574643f\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" Nov 01 06:41:53.604152 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.604104 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcvmt\" (UniqueName: \"kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-kube-api-access-tcvmt\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc\" (UID: \"e5a2a7f9-63f8-44ae-a226-c5fd9574643f\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" Nov 01 06:41:53.604267 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.604245 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc\" (UID: \"e5a2a7f9-63f8-44ae-a226-c5fd9574643f\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" Nov 01 06:41:53.604338 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.604318 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc\" (UID: \"e5a2a7f9-63f8-44ae-a226-c5fd9574643f\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" Nov 01 06:41:53.652207 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.652175 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"kube-root-ca.crt\"" Nov 01 06:41:53.652469 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.652445 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"agc-cred-c3e56f59453b\"" Nov 01 06:41:53.652546 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.652445 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"agc-cred-841de3b011bd\"" Nov 01 06:41:53.652546 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.652484 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"agc-cred-529628bc0fc6\"" Nov 01 06:41:53.652546 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.652495 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"openshift-service-ca.crt\"" Nov 01 06:41:53.652546 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.652536 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"routing-overlay\"" Nov 01 06:41:53.652713 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.652562 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"agc-cred-97c3302b7889\"" Nov 01 06:41:53.652713 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.652575 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"payload-processing-external-model-e2e--528f7081-dockercfg-c6b6l\"" Nov 01 06:41:53.705787 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.705748 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc\" (UID: \"e5a2a7f9-63f8-44ae-a226-c5fd9574643f\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" Nov 01 06:41:53.706034 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.705818 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcvmt\" (UniqueName: \"kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-kube-api-access-tcvmt\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc\" (UID: \"e5a2a7f9-63f8-44ae-a226-c5fd9574643f\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" Nov 01 06:41:53.706034 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.705889 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc\" (UID: \"e5a2a7f9-63f8-44ae-a226-c5fd9574643f\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" Nov 01 06:41:53.706034 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.705940 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc\" (UID: \"e5a2a7f9-63f8-44ae-a226-c5fd9574643f\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" Nov 01 06:41:53.706617 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.706553 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc\" (UID: \"e5a2a7f9-63f8-44ae-a226-c5fd9574643f\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" Nov 01 06:41:53.706895 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.706870 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc\" (UID: \"e5a2a7f9-63f8-44ae-a226-c5fd9574643f\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" Nov 01 06:41:53.708727 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.708706 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc\" (UID: \"e5a2a7f9-63f8-44ae-a226-c5fd9574643f\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" Nov 01 06:41:53.762788 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.762753 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcvmt\" (UniqueName: \"kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-kube-api-access-tcvmt\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc\" (UID: \"e5a2a7f9-63f8-44ae-a226-c5fd9574643f\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" Nov 01 06:41:53.871490 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:53.871457 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" Nov 01 06:41:57.260211 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:57.260174 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:57.427341 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:57.427309 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:57.584201 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:57.584158 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:58.646549 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:58.646517 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:41:58.854337 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:41:58.854303 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:02.221549 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:02.221461 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:05.828273 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:05.828236 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:06.780033 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:42:06.779989 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:42:07.136010 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:07.135972 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:11.146944 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:11.146908 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:11.303715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:11.303681 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:11.751459 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:11.751431 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:11.944593 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:11.944559 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:12.152144 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:12.152107 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:14.543990 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.543959 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz"] Nov 01 06:42:14.548251 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.548231 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" Nov 01 06:42:14.600759 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.600719 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"payload-processing-external-model-plugins-e2e-extmod-b-dc6e48\"" Nov 01 06:42:14.600759 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.600734 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-c8aede0ff036\"" Nov 01 06:42:14.600759 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.600766 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-a83cd424b5a5\"" Nov 01 06:42:14.601089 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.600822 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"kube-root-ca.crt\"" Nov 01 06:42:14.650774 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.650741 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-120370fe39d3\"" Nov 01 06:42:14.651015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.650753 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"routing-overlay\"" Nov 01 06:42:14.651015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.650752 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-2b38f786e9be\"" Nov 01 06:42:14.651015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.650770 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"openshift-service-ca.crt\"" Nov 01 06:42:14.651015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.650769 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"payload-processing-external-model-e2e--958558de-dockercfg-qjzd6\"" Nov 01 06:42:14.732228 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.732195 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/377165c8-40b7-4fe6-a362-884eb9712662-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz\" (UID: \"377165c8-40b7-4fe6-a362-884eb9712662\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" Nov 01 06:42:14.732409 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.732254 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/377165c8-40b7-4fe6-a362-884eb9712662-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz\" (UID: \"377165c8-40b7-4fe6-a362-884eb9712662\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" Nov 01 06:42:14.732409 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.732363 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbmd9\" (UniqueName: \"kubernetes.io/projected/377165c8-40b7-4fe6-a362-884eb9712662-kube-api-access-pbmd9\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz\" (UID: \"377165c8-40b7-4fe6-a362-884eb9712662\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" Nov 01 06:42:14.732487 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.732424 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/377165c8-40b7-4fe6-a362-884eb9712662-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz\" (UID: \"377165c8-40b7-4fe6-a362-884eb9712662\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" Nov 01 06:42:14.833714 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.833677 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/377165c8-40b7-4fe6-a362-884eb9712662-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz\" (UID: \"377165c8-40b7-4fe6-a362-884eb9712662\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" Nov 01 06:42:14.833921 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.833757 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/377165c8-40b7-4fe6-a362-884eb9712662-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz\" (UID: \"377165c8-40b7-4fe6-a362-884eb9712662\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" Nov 01 06:42:14.833921 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.833805 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbmd9\" (UniqueName: \"kubernetes.io/projected/377165c8-40b7-4fe6-a362-884eb9712662-kube-api-access-pbmd9\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz\" (UID: \"377165c8-40b7-4fe6-a362-884eb9712662\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" Nov 01 06:42:14.834000 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.833931 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/377165c8-40b7-4fe6-a362-884eb9712662-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz\" (UID: \"377165c8-40b7-4fe6-a362-884eb9712662\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" Nov 01 06:42:14.834520 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.834486 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/377165c8-40b7-4fe6-a362-884eb9712662-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz\" (UID: \"377165c8-40b7-4fe6-a362-884eb9712662\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" Nov 01 06:42:14.834520 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.834508 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/377165c8-40b7-4fe6-a362-884eb9712662-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz\" (UID: \"377165c8-40b7-4fe6-a362-884eb9712662\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" Nov 01 06:42:14.836497 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.836476 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/377165c8-40b7-4fe6-a362-884eb9712662-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz\" (UID: \"377165c8-40b7-4fe6-a362-884eb9712662\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" Nov 01 06:42:14.890208 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:14.890181 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbmd9\" (UniqueName: \"kubernetes.io/projected/377165c8-40b7-4fe6-a362-884eb9712662-kube-api-access-pbmd9\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz\" (UID: \"377165c8-40b7-4fe6-a362-884eb9712662\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" Nov 01 06:42:15.162577 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:15.162488 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" Nov 01 06:42:16.633204 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:16.633147 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:17.598587 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:17.598548 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:17.751703 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:17.751666 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:21.324229 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:21.324195 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:21.524011 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:21.523973 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:23.088364 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:23.088324 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:23.240772 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:23.240722 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:23.760710 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:23.760678 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:24.039068 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:24.038977 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:24.323054 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:24.322947 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz"] Nov 01 06:42:24.326149 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:24.326117 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54"] Nov 01 06:42:24.331317 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:24.331291 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" Nov 01 06:42:24.445107 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:24.445070 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dcd72965-a94b-4f78-bd08-8a35bbd62b40-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54\" (UID: \"dcd72965-a94b-4f78-bd08-8a35bbd62b40\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" Nov 01 06:42:24.445302 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:24.445201 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dcd72965-a94b-4f78-bd08-8a35bbd62b40-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54\" (UID: \"dcd72965-a94b-4f78-bd08-8a35bbd62b40\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" Nov 01 06:42:24.445302 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:24.445261 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcd72965-a94b-4f78-bd08-8a35bbd62b40-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54\" (UID: \"dcd72965-a94b-4f78-bd08-8a35bbd62b40\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" Nov 01 06:42:24.445483 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:24.445299 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7xnc\" (UniqueName: \"kubernetes.io/projected/dcd72965-a94b-4f78-bd08-8a35bbd62b40-kube-api-access-z7xnc\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54\" (UID: \"dcd72965-a94b-4f78-bd08-8a35bbd62b40\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" Nov 01 06:42:24.546541 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:24.546497 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dcd72965-a94b-4f78-bd08-8a35bbd62b40-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54\" (UID: \"dcd72965-a94b-4f78-bd08-8a35bbd62b40\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" Nov 01 06:42:24.546763 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:24.546569 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dcd72965-a94b-4f78-bd08-8a35bbd62b40-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54\" (UID: \"dcd72965-a94b-4f78-bd08-8a35bbd62b40\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" Nov 01 06:42:24.546763 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:24.546600 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcd72965-a94b-4f78-bd08-8a35bbd62b40-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54\" (UID: \"dcd72965-a94b-4f78-bd08-8a35bbd62b40\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" Nov 01 06:42:24.546763 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:24.546707 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7xnc\" (UniqueName: \"kubernetes.io/projected/dcd72965-a94b-4f78-bd08-8a35bbd62b40-kube-api-access-z7xnc\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54\" (UID: \"dcd72965-a94b-4f78-bd08-8a35bbd62b40\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" Nov 01 06:42:24.547452 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:24.547427 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dcd72965-a94b-4f78-bd08-8a35bbd62b40-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54\" (UID: \"dcd72965-a94b-4f78-bd08-8a35bbd62b40\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" Nov 01 06:42:24.547540 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:24.547438 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcd72965-a94b-4f78-bd08-8a35bbd62b40-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54\" (UID: \"dcd72965-a94b-4f78-bd08-8a35bbd62b40\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" Nov 01 06:42:24.549620 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:24.549596 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dcd72965-a94b-4f78-bd08-8a35bbd62b40-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54\" (UID: \"dcd72965-a94b-4f78-bd08-8a35bbd62b40\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" Nov 01 06:42:24.604040 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:24.604003 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7xnc\" (UniqueName: \"kubernetes.io/projected/dcd72965-a94b-4f78-bd08-8a35bbd62b40-kube-api-access-z7xnc\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54\" (UID: \"dcd72965-a94b-4f78-bd08-8a35bbd62b40\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" Nov 01 06:42:24.646126 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:24.646096 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" Nov 01 06:42:26.146323 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:26.146285 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:27.231894 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:27.231857 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:29.750238 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:29.750204 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:29.952067 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:29.952024 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:30.186120 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:30.186087 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:30.811724 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:30.811685 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54"] Nov 01 06:42:30.816039 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:30.816013 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk"] Nov 01 06:42:30.843867 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:30.843797 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:30.860306 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:30.860274 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc"] Nov 01 06:42:30.863390 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:30.863373 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz"] Nov 01 06:42:31.207757 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:31.207712 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:31.313813 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:31.313784 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54"] Nov 01 06:42:31.314477 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:42:31.314454 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd72965_a94b_4f78_bd08_8a35bbd62b40.slice/crio-dcf033472cecdf57cea324d5fe3d84b941c77d76249dbb431cc6fd0a9d3f22d9 WatchSource:0}: Error finding container dcf033472cecdf57cea324d5fe3d84b941c77d76249dbb431cc6fd0a9d3f22d9: Status 404 returned error can't find the container with id dcf033472cecdf57cea324d5fe3d84b941c77d76249dbb431cc6fd0a9d3f22d9 Nov 01 06:42:31.362945 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:31.362916 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz"] Nov 01 06:42:31.367206 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:31.367177 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:31.368473 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:31.367975 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc"] Nov 01 06:42:31.371336 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:31.371314 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk"] Nov 01 06:42:31.384129 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:42:31.381126 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377165c8_40b7_4fe6_a362_884eb9712662.slice/crio-c73931b0ba5981e84481364ab4b667fae1ad1bea80a254334cb2006d0c71f4d6 WatchSource:0}: Error finding container c73931b0ba5981e84481364ab4b667fae1ad1bea80a254334cb2006d0c71f4d6: Status 404 returned error can't find the container with id c73931b0ba5981e84481364ab4b667fae1ad1bea80a254334cb2006d0c71f4d6 Nov 01 06:42:31.752536 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:31.752443 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" event={"ID":"e5a2a7f9-63f8-44ae-a226-c5fd9574643f","Type":"ContainerStarted","Data":"dc1def466425333fe816195caba301eb35ced9314c1307d5f0311bdae5c2c61e"} Nov 01 06:42:31.752536 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:31.752512 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" event={"ID":"e5a2a7f9-63f8-44ae-a226-c5fd9574643f","Type":"ContainerStarted","Data":"c1a18ab2131b6a54e8e0b74c9b3914825d0735a82d39e05f3272f5173f51e328"} Nov 01 06:42:31.752741 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:31.752561 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" Nov 01 06:42:31.753987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:31.753964 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" event={"ID":"377165c8-40b7-4fe6-a362-884eb9712662","Type":"ContainerStarted","Data":"36841b79b4b148df9d4fbf88d8fdf17590e7042bbc3ab7e86baa0c57a4f8d692"} Nov 01 06:42:31.754137 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:31.754002 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" event={"ID":"377165c8-40b7-4fe6-a362-884eb9712662","Type":"ContainerStarted","Data":"c73931b0ba5981e84481364ab4b667fae1ad1bea80a254334cb2006d0c71f4d6"} Nov 01 06:42:31.754137 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:31.754054 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" Nov 01 06:42:31.755539 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:31.755520 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" event={"ID":"46bcb2e8-bdb2-4632-911d-d0668846115a","Type":"ContainerStarted","Data":"306a397e97f18fc4d7442a0b6fc1507014fcfab883495394435f0999b87cd006"} Nov 01 06:42:31.755634 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:31.755556 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" event={"ID":"46bcb2e8-bdb2-4632-911d-d0668846115a","Type":"ContainerStarted","Data":"e037ba3bab900596a68e3b643dd4be82d41f0cf218e0ae69a6d3cbe4bf445c60"} Nov 01 06:42:31.755634 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:31.755582 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" Nov 01 06:42:31.757063 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:31.757045 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" event={"ID":"dcd72965-a94b-4f78-bd08-8a35bbd62b40","Type":"ContainerStarted","Data":"1eb08c9fb6e6a9d9650636e25fe62e1ba12291afb2be37680894088483c2ceff"} Nov 01 06:42:31.757155 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:31.757082 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" event={"ID":"dcd72965-a94b-4f78-bd08-8a35bbd62b40","Type":"ContainerStarted","Data":"dcf033472cecdf57cea324d5fe3d84b941c77d76249dbb431cc6fd0a9d3f22d9"} Nov 01 06:42:31.757155 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:31.757143 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" Nov 01 06:42:31.867311 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:31.867263 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" podStartSLOduration=38.867247732 podStartE2EDuration="38.867247732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:42:31.864341434 +0000 UTC m=+1961.628141293" watchObservedRunningTime="2026-11-01 06:42:31.867247732 +0000 UTC m=+1961.631047589" Nov 01 06:42:31.978593 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:31.978542 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" podStartSLOduration=7.978528207 podStartE2EDuration="7.978528207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:42:31.976988197 +0000 UTC m=+1961.740788055" watchObservedRunningTime="2026-11-01 06:42:31.978528207 +0000 UTC m=+1961.742328073" Nov 01 06:42:32.091001 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:32.090949 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" podStartSLOduration=41.090935165 podStartE2EDuration="41.090935165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:42:32.088868407 +0000 UTC m=+1961.852668263" watchObservedRunningTime="2026-11-01 06:42:32.090935165 +0000 UTC m=+1961.854735023" Nov 01 06:42:32.204140 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:32.204087 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" podStartSLOduration=18.204072302 podStartE2EDuration="18.204072302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:42:32.201716328 +0000 UTC m=+1961.965516182" watchObservedRunningTime="2026-11-01 06:42:32.204072302 +0000 UTC m=+1961.967872198" Nov 01 06:42:35.316721 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:35.316681 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54"] Nov 01 06:42:35.637453 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:35.637410 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:36.755138 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:36.755094 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" podUID="377165c8-40b7-4fe6-a362-884eb9712662" containerName="payload-processing" containerID="cri-o://36841b79b4b148df9d4fbf88d8fdf17590e7042bbc3ab7e86baa0c57a4f8d692" gracePeriod=25 Nov 01 06:42:36.780387 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:42:36.780350 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:42:37.020666 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:37.020601 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" Nov 01 06:42:37.077951 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:37.077914 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377165c8-40b7-4fe6-a362-884eb9712662-provider-credentials\" (UniqueName: \"kubernetes.io/projected/377165c8-40b7-4fe6-a362-884eb9712662-provider-credentials\") pod \"377165c8-40b7-4fe6-a362-884eb9712662\" (UID: \"377165c8-40b7-4fe6-a362-884eb9712662\") " Nov 01 06:42:37.078163 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:37.078007 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/377165c8-40b7-4fe6-a362-884eb9712662-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/377165c8-40b7-4fe6-a362-884eb9712662-plugins-config-volume\") pod \"377165c8-40b7-4fe6-a362-884eb9712662\" (UID: \"377165c8-40b7-4fe6-a362-884eb9712662\") " Nov 01 06:42:37.078163 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:37.078067 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/377165c8-40b7-4fe6-a362-884eb9712662-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/377165c8-40b7-4fe6-a362-884eb9712662-routing-overlay-volume\") pod \"377165c8-40b7-4fe6-a362-884eb9712662\" (UID: \"377165c8-40b7-4fe6-a362-884eb9712662\") " Nov 01 06:42:37.078163 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:37.078143 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377165c8-40b7-4fe6-a362-884eb9712662-kube-api-access-pbmd9\" (UniqueName: \"kubernetes.io/projected/377165c8-40b7-4fe6-a362-884eb9712662-kube-api-access-pbmd9\") pod \"377165c8-40b7-4fe6-a362-884eb9712662\" (UID: \"377165c8-40b7-4fe6-a362-884eb9712662\") " Nov 01 06:42:37.078471 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:37.078446 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/377165c8-40b7-4fe6-a362-884eb9712662-routing-overlay-volume" pod "377165c8-40b7-4fe6-a362-884eb9712662" (UID: "377165c8-40b7-4fe6-a362-884eb9712662"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:42:37.078471 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:37.078463 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/377165c8-40b7-4fe6-a362-884eb9712662-plugins-config-volume" pod "377165c8-40b7-4fe6-a362-884eb9712662" (UID: "377165c8-40b7-4fe6-a362-884eb9712662"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:42:37.080392 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:37.080369 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377165c8-40b7-4fe6-a362-884eb9712662-kube-api-access-pbmd9" pod "377165c8-40b7-4fe6-a362-884eb9712662" (UID: "377165c8-40b7-4fe6-a362-884eb9712662"). InnerVolumeSpecName "kube-api-access-pbmd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:42:37.080488 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:37.080399 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377165c8-40b7-4fe6-a362-884eb9712662-provider-credentials" pod "377165c8-40b7-4fe6-a362-884eb9712662" (UID: "377165c8-40b7-4fe6-a362-884eb9712662"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:42:37.179828 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:37.179792 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbmd9\" (UniqueName: \"kubernetes.io/projected/377165c8-40b7-4fe6-a362-884eb9712662-kube-api-access-pbmd9\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:37.180011 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:37.179825 4385 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/377165c8-40b7-4fe6-a362-884eb9712662-provider-credentials\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:37.180011 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:37.179865 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/377165c8-40b7-4fe6-a362-884eb9712662-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:37.180011 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:37.179874 4385 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/377165c8-40b7-4fe6-a362-884eb9712662-routing-overlay-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:37.800395 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:37.800361 4385 generic.go:356] "Generic (PLEG): container finished" podID="377165c8-40b7-4fe6-a362-884eb9712662" containerID="36841b79b4b148df9d4fbf88d8fdf17590e7042bbc3ab7e86baa0c57a4f8d692" exitCode=0 Nov 01 06:42:37.800945 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:37.800402 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" Nov 01 06:42:37.800945 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:37.800450 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" event={"ID":"377165c8-40b7-4fe6-a362-884eb9712662","Type":"ContainerDied","Data":"36841b79b4b148df9d4fbf88d8fdf17590e7042bbc3ab7e86baa0c57a4f8d692"} Nov 01 06:42:37.800945 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:37.800503 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz" event={"ID":"377165c8-40b7-4fe6-a362-884eb9712662","Type":"ContainerDied","Data":"c73931b0ba5981e84481364ab4b667fae1ad1bea80a254334cb2006d0c71f4d6"} Nov 01 06:42:37.800945 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:37.800527 4385 scope.go:122] "RemoveContainer" containerID="36841b79b4b148df9d4fbf88d8fdf17590e7042bbc3ab7e86baa0c57a4f8d692" Nov 01 06:42:37.811611 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:37.811589 4385 scope.go:122] "RemoveContainer" containerID="36841b79b4b148df9d4fbf88d8fdf17590e7042bbc3ab7e86baa0c57a4f8d692" Nov 01 06:42:37.811995 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:42:37.811966 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36841b79b4b148df9d4fbf88d8fdf17590e7042bbc3ab7e86baa0c57a4f8d692\": container with ID starting with 36841b79b4b148df9d4fbf88d8fdf17590e7042bbc3ab7e86baa0c57a4f8d692 not found: ID does not exist" containerID="36841b79b4b148df9d4fbf88d8fdf17590e7042bbc3ab7e86baa0c57a4f8d692" Nov 01 06:42:37.812099 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:37.812004 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36841b79b4b148df9d4fbf88d8fdf17590e7042bbc3ab7e86baa0c57a4f8d692"} err="failed to get container status \"36841b79b4b148df9d4fbf88d8fdf17590e7042bbc3ab7e86baa0c57a4f8d692\": rpc error: code = NotFound desc = could not find container \"36841b79b4b148df9d4fbf88d8fdf17590e7042bbc3ab7e86baa0c57a4f8d692\": container with ID starting with 36841b79b4b148df9d4fbf88d8fdf17590e7042bbc3ab7e86baa0c57a4f8d692 not found: ID does not exist" Nov 01 06:42:37.964198 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:37.964161 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz"] Nov 01 06:42:38.013750 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:38.013719 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-5c4d9clfz"] Nov 01 06:42:38.384515 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:38.384478 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="377165c8-40b7-4fe6-a362-884eb9712662" path="/var/lib/kubelet/pods/377165c8-40b7-4fe6-a362-884eb9712662/volumes" Nov 01 06:42:39.283801 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.283766 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w"] Nov 01 06:42:39.284304 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.284289 4385 cpu_manager.go:414] "containerMap: removing container" podUID="377165c8-40b7-4fe6-a362-884eb9712662" containerName="payload-processing" Nov 01 06:42:39.284354 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.284308 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377165c8-40b7-4fe6-a362-884eb9712662" containerName="payload-processing" Nov 01 06:42:39.284428 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.284419 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="377165c8-40b7-4fe6-a362-884eb9712662" containerName="payload-processing" Nov 01 06:42:39.289216 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.289193 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.395237 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.395206 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w"] Nov 01 06:42:39.404978 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.404949 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25f4c33e-6491-4057-8979-f8ff03d56c6f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.405147 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.405006 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.405147 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.405126 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79qj4\" (UniqueName: \"kubernetes.io/projected/25f4c33e-6491-4057-8979-f8ff03d56c6f-kube-api-access-79qj4\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.405238 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.405177 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.405238 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.405217 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.405320 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.405292 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.405320 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.405313 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.405390 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.405353 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.405444 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.405422 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.506239 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.506199 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.506435 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.506251 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.506435 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.506395 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25f4c33e-6491-4057-8979-f8ff03d56c6f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.506552 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.506450 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.506552 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.506487 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79qj4\" (UniqueName: \"kubernetes.io/projected/25f4c33e-6491-4057-8979-f8ff03d56c6f-kube-api-access-79qj4\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.506552 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.506520 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.506552 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.506544 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.506773 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.506597 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.506773 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.506628 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.506773 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.506688 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.507074 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.507030 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.507145 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.507135 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.507205 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.507187 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.507274 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.507254 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25f4c33e-6491-4057-8979-f8ff03d56c6f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.509323 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.509295 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.509698 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.509674 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.563257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.563177 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.563425 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.563290 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79qj4\" (UniqueName: \"kubernetes.io/projected/25f4c33e-6491-4057-8979-f8ff03d56c6f-kube-api-access-79qj4\") pod \"maas-default-gateway-openshift-default-7d9897b54c-njl8w\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:39.603986 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:39.603947 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:40.038806 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.038777 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w"] Nov 01 06:42:40.039294 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:42:40.039271 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f4c33e_6491_4057_8979_f8ff03d56c6f.slice/crio-95c7a9186dfd807776aab1d588f37af9333ea6474e1240d2240e7365bee93d2b WatchSource:0}: Error finding container 95c7a9186dfd807776aab1d588f37af9333ea6474e1240d2240e7365bee93d2b: Status 404 returned error can't find the container with id 95c7a9186dfd807776aab1d588f37af9333ea6474e1240d2240e7365bee93d2b Nov 01 06:42:40.042273 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.042239 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:42:40.042350 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.042311 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:42:40.042350 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.042344 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:42:40.317383 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.317244 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" podUID="dcd72965-a94b-4f78-bd08-8a35bbd62b40" containerName="payload-processing" containerID="cri-o://1eb08c9fb6e6a9d9650636e25fe62e1ba12291afb2be37680894088483c2ceff" gracePeriod=25 Nov 01 06:42:40.323306 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.323279 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:40.595876 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.595823 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" Nov 01 06:42:40.722557 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.722520 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcd72965-a94b-4f78-bd08-8a35bbd62b40-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dcd72965-a94b-4f78-bd08-8a35bbd62b40-routing-overlay-volume\") pod \"dcd72965-a94b-4f78-bd08-8a35bbd62b40\" (UID: \"dcd72965-a94b-4f78-bd08-8a35bbd62b40\") " Nov 01 06:42:40.722747 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.722580 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcd72965-a94b-4f78-bd08-8a35bbd62b40-kube-api-access-z7xnc\" (UniqueName: \"kubernetes.io/projected/dcd72965-a94b-4f78-bd08-8a35bbd62b40-kube-api-access-z7xnc\") pod \"dcd72965-a94b-4f78-bd08-8a35bbd62b40\" (UID: \"dcd72965-a94b-4f78-bd08-8a35bbd62b40\") " Nov 01 06:42:40.722747 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.722720 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcd72965-a94b-4f78-bd08-8a35bbd62b40-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dcd72965-a94b-4f78-bd08-8a35bbd62b40-provider-credentials\") pod \"dcd72965-a94b-4f78-bd08-8a35bbd62b40\" (UID: \"dcd72965-a94b-4f78-bd08-8a35bbd62b40\") " Nov 01 06:42:40.722938 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.722759 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcd72965-a94b-4f78-bd08-8a35bbd62b40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcd72965-a94b-4f78-bd08-8a35bbd62b40-plugins-config-volume\") pod \"dcd72965-a94b-4f78-bd08-8a35bbd62b40\" (UID: \"dcd72965-a94b-4f78-bd08-8a35bbd62b40\") " Nov 01 06:42:40.722938 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.722922 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd72965-a94b-4f78-bd08-8a35bbd62b40-routing-overlay-volume" pod "dcd72965-a94b-4f78-bd08-8a35bbd62b40" (UID: "dcd72965-a94b-4f78-bd08-8a35bbd62b40"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:42:40.723288 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.723266 4385 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dcd72965-a94b-4f78-bd08-8a35bbd62b40-routing-overlay-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:40.723383 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.723305 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd72965-a94b-4f78-bd08-8a35bbd62b40-plugins-config-volume" pod "dcd72965-a94b-4f78-bd08-8a35bbd62b40" (UID: "dcd72965-a94b-4f78-bd08-8a35bbd62b40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:42:40.725376 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.725349 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd72965-a94b-4f78-bd08-8a35bbd62b40-provider-credentials" pod "dcd72965-a94b-4f78-bd08-8a35bbd62b40" (UID: "dcd72965-a94b-4f78-bd08-8a35bbd62b40"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:42:40.725376 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.725363 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd72965-a94b-4f78-bd08-8a35bbd62b40-kube-api-access-z7xnc" pod "dcd72965-a94b-4f78-bd08-8a35bbd62b40" (UID: "dcd72965-a94b-4f78-bd08-8a35bbd62b40"). InnerVolumeSpecName "kube-api-access-z7xnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:42:40.826143 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.826106 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7xnc\" (UniqueName: \"kubernetes.io/projected/dcd72965-a94b-4f78-bd08-8a35bbd62b40-kube-api-access-z7xnc\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:40.826362 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.826153 4385 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dcd72965-a94b-4f78-bd08-8a35bbd62b40-provider-credentials\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:40.826362 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.826171 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcd72965-a94b-4f78-bd08-8a35bbd62b40-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:40.831529 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.831486 4385 generic.go:356] "Generic (PLEG): container finished" podID="dcd72965-a94b-4f78-bd08-8a35bbd62b40" containerID="1eb08c9fb6e6a9d9650636e25fe62e1ba12291afb2be37680894088483c2ceff" exitCode=0 Nov 01 06:42:40.831737 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.831709 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" event={"ID":"dcd72965-a94b-4f78-bd08-8a35bbd62b40","Type":"ContainerDied","Data":"1eb08c9fb6e6a9d9650636e25fe62e1ba12291afb2be37680894088483c2ceff"} Nov 01 06:42:40.831855 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.831732 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" Nov 01 06:42:40.832002 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.831979 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54" event={"ID":"dcd72965-a94b-4f78-bd08-8a35bbd62b40","Type":"ContainerDied","Data":"dcf033472cecdf57cea324d5fe3d84b941c77d76249dbb431cc6fd0a9d3f22d9"} Nov 01 06:42:40.832091 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.832037 4385 scope.go:122] "RemoveContainer" containerID="1eb08c9fb6e6a9d9650636e25fe62e1ba12291afb2be37680894088483c2ceff" Nov 01 06:42:40.834116 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.834094 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" event={"ID":"25f4c33e-6491-4057-8979-f8ff03d56c6f","Type":"ContainerStarted","Data":"16a61ffe1da8eb241d86ff3014592dae5a02633ae16c21984275132ab187fd3c"} Nov 01 06:42:40.834203 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.834155 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" event={"ID":"25f4c33e-6491-4057-8979-f8ff03d56c6f","Type":"ContainerStarted","Data":"95c7a9186dfd807776aab1d588f37af9333ea6474e1240d2240e7365bee93d2b"} Nov 01 06:42:40.845250 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.845228 4385 scope.go:122] "RemoveContainer" containerID="1eb08c9fb6e6a9d9650636e25fe62e1ba12291afb2be37680894088483c2ceff" Nov 01 06:42:40.845598 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:42:40.845561 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eb08c9fb6e6a9d9650636e25fe62e1ba12291afb2be37680894088483c2ceff\": container with ID starting with 1eb08c9fb6e6a9d9650636e25fe62e1ba12291afb2be37680894088483c2ceff not found: ID does not exist" containerID="1eb08c9fb6e6a9d9650636e25fe62e1ba12291afb2be37680894088483c2ceff" Nov 01 06:42:40.845646 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:40.845609 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb08c9fb6e6a9d9650636e25fe62e1ba12291afb2be37680894088483c2ceff"} err="failed to get container status \"1eb08c9fb6e6a9d9650636e25fe62e1ba12291afb2be37680894088483c2ceff\": rpc error: code = NotFound desc = could not find container \"1eb08c9fb6e6a9d9650636e25fe62e1ba12291afb2be37680894088483c2ceff\": container with ID starting with 1eb08c9fb6e6a9d9650636e25fe62e1ba12291afb2be37680894088483c2ceff not found: ID does not exist" Nov 01 06:42:41.003504 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:41.003466 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54"] Nov 01 06:42:41.009032 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:41.008999 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5gqp54"] Nov 01 06:42:41.119899 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:41.119775 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" podStartSLOduration=2.119759825 podStartE2EDuration="2.119759825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:42:41.116996216 +0000 UTC m=+1970.880796072" watchObservedRunningTime="2026-11-01 06:42:41.119759825 +0000 UTC m=+1970.883559681" Nov 01 06:42:41.604205 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:41.604173 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:41.605943 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:41.605914 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-njl8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 06:42:41.606110 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:41.605975 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" podUID="25f4c33e-6491-4057-8979-f8ff03d56c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 06:42:42.406796 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:42.406747 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcd72965-a94b-4f78-bd08-8a35bbd62b40" path="/var/lib/kubelet/pods/dcd72965-a94b-4f78-bd08-8a35bbd62b40/volumes" Nov 01 06:42:42.604607 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:42.604564 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-njl8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 06:42:42.605118 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:42.604640 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" podUID="25f4c33e-6491-4057-8979-f8ff03d56c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 06:42:42.766710 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:42.766637 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" Nov 01 06:42:42.766710 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:42.766710 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" Nov 01 06:42:43.405541 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:43.405508 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:43.606922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:43.604536 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-njl8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 06:42:43.606922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:43.604613 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" podUID="25f4c33e-6491-4057-8979-f8ff03d56c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 06:42:44.604854 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:44.604803 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-njl8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 06:42:44.605126 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:44.605100 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" podUID="25f4c33e-6491-4057-8979-f8ff03d56c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 06:42:45.003859 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:45.003617 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:45.151504 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:45.151469 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:45.346783 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:45.346728 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:45.606042 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:45.605125 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-njl8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 06:42:45.606042 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:45.605175 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" podUID="25f4c33e-6491-4057-8979-f8ff03d56c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 06:42:46.605149 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:46.605110 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-njl8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 06:42:46.605675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:46.605192 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" podUID="25f4c33e-6491-4057-8979-f8ff03d56c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 06:42:47.604752 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:47.604709 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-njl8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 06:42:47.604986 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:47.604792 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" podUID="25f4c33e-6491-4057-8979-f8ff03d56c6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 06:42:48.959531 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:48.959482 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:48.960100 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:48.959951 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:48.961037 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:48.961018 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 06:42:49.924197 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:49.924160 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:54.412385 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.412352 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk"] Nov 01 06:42:54.413001 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.412827 4385 cpu_manager.go:414] "containerMap: removing container" podUID="dcd72965-a94b-4f78-bd08-8a35bbd62b40" containerName="payload-processing" Nov 01 06:42:54.413001 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.412852 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcd72965-a94b-4f78-bd08-8a35bbd62b40" containerName="payload-processing" Nov 01 06:42:54.413001 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.412984 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcd72965-a94b-4f78-bd08-8a35bbd62b40" containerName="payload-processing" Nov 01 06:42:54.417364 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.417338 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.469792 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.469761 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3893a1b-d9b9-432b-ac25-28df30013563-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.469989 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.469858 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3893a1b-d9b9-432b-ac25-28df30013563-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.470054 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.470023 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3893a1b-d9b9-432b-ac25-28df30013563-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.470101 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.470091 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3893a1b-d9b9-432b-ac25-28df30013563-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.470217 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.470196 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3893a1b-d9b9-432b-ac25-28df30013563-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.470299 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.470250 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99qc4\" (UniqueName: \"kubernetes.io/projected/f3893a1b-d9b9-432b-ac25-28df30013563-kube-api-access-99qc4\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.470357 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.470338 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3893a1b-d9b9-432b-ac25-28df30013563-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.470451 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.470428 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3893a1b-d9b9-432b-ac25-28df30013563-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.470522 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.470485 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3893a1b-d9b9-432b-ac25-28df30013563-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.524748 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.524707 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk"] Nov 01 06:42:54.571632 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.571586 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3893a1b-d9b9-432b-ac25-28df30013563-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.571879 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.571688 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3893a1b-d9b9-432b-ac25-28df30013563-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.571879 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.571727 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3893a1b-d9b9-432b-ac25-28df30013563-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.571879 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.571762 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3893a1b-d9b9-432b-ac25-28df30013563-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.571879 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.571855 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3893a1b-d9b9-432b-ac25-28df30013563-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.572141 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.571889 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99qc4\" (UniqueName: \"kubernetes.io/projected/f3893a1b-d9b9-432b-ac25-28df30013563-kube-api-access-99qc4\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.572141 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.571949 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3893a1b-d9b9-432b-ac25-28df30013563-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.572141 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.571977 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3893a1b-d9b9-432b-ac25-28df30013563-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.572141 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.572014 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3893a1b-d9b9-432b-ac25-28df30013563-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.572377 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.572350 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3893a1b-d9b9-432b-ac25-28df30013563-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.572536 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.572513 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3893a1b-d9b9-432b-ac25-28df30013563-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.572615 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.572551 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3893a1b-d9b9-432b-ac25-28df30013563-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.572615 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.572599 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3893a1b-d9b9-432b-ac25-28df30013563-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.573031 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.573006 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3893a1b-d9b9-432b-ac25-28df30013563-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.574990 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.574966 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3893a1b-d9b9-432b-ac25-28df30013563-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.575078 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.574987 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3893a1b-d9b9-432b-ac25-28df30013563-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.628213 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.628184 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3893a1b-d9b9-432b-ac25-28df30013563-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.628469 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.628448 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99qc4\" (UniqueName: \"kubernetes.io/projected/f3893a1b-d9b9-432b-ac25-28df30013563-kube-api-access-99qc4\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7ldgk\" (UID: \"f3893a1b-d9b9-432b-ac25-28df30013563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.731201 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.731100 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:54.740567 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:54.740528 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:55.173292 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:55.173263 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk"] Nov 01 06:42:55.173556 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:42:55.173532 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3893a1b_d9b9_432b_ac25_28df30013563.slice/crio-28f15f0f2f44a8421b704a0e9a599e4ce2911e0ada18e43ef4a1e27120de0fa7 WatchSource:0}: Error finding container 28f15f0f2f44a8421b704a0e9a599e4ce2911e0ada18e43ef4a1e27120de0fa7: Status 404 returned error can't find the container with id 28f15f0f2f44a8421b704a0e9a599e4ce2911e0ada18e43ef4a1e27120de0fa7 Nov 01 06:42:55.176002 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:55.175971 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:42:55.176103 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:55.176033 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:42:55.176103 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:55.176063 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:42:55.961361 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:55.961325 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" event={"ID":"f3893a1b-d9b9-432b-ac25-28df30013563","Type":"ContainerStarted","Data":"be18893f1154d6a72cb60bad3f6c9b3e5ab3b219fe7e98fa704ef0d68d8229b1"} Nov 01 06:42:55.961816 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:55.961382 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" event={"ID":"f3893a1b-d9b9-432b-ac25-28df30013563","Type":"ContainerStarted","Data":"28f15f0f2f44a8421b704a0e9a599e4ce2911e0ada18e43ef4a1e27120de0fa7"} Nov 01 06:42:56.091859 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:56.091314 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" podStartSLOduration=2.091291879 podStartE2EDuration="2.091291879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:42:56.088365852 +0000 UTC m=+1985.852165748" watchObservedRunningTime="2026-11-01 06:42:56.091291879 +0000 UTC m=+1985.855091736" Nov 01 06:42:56.731677 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:56.731647 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:42:56.733440 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:56.733414 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7ldgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 06:42:56.733562 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:56.733465 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" podUID="f3893a1b-d9b9-432b-ac25-28df30013563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 06:42:57.731680 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:57.731643 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7ldgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 06:42:57.732128 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:57.731708 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" podUID="f3893a1b-d9b9-432b-ac25-28df30013563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 06:42:58.732493 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:58.732449 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7ldgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 06:42:58.732988 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:58.732514 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" podUID="f3893a1b-d9b9-432b-ac25-28df30013563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 06:42:59.416106 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:59.416072 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:42:59.731740 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:59.731654 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7ldgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 06:42:59.731740 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:42:59.731709 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" podUID="f3893a1b-d9b9-432b-ac25-28df30013563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 06:43:00.732121 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:00.732087 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7ldgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 06:43:00.732549 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:00.732151 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" podUID="f3893a1b-d9b9-432b-ac25-28df30013563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 06:43:01.731822 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:01.731780 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7ldgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 06:43:01.732084 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:01.731896 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" podUID="f3893a1b-d9b9-432b-ac25-28df30013563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 06:43:02.731844 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:02.731794 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7ldgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 06:43:02.732316 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:02.731864 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" podUID="f3893a1b-d9b9-432b-ac25-28df30013563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 06:43:03.757846 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:03.757802 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:43:03.758959 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:03.758935 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:43:03.759560 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:03.759536 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" Nov 01 06:43:04.124110 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:04.124080 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:06.780910 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:43:06.780871 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:43:08.830901 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:08.830860 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:13.565471 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:13.565432 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:18.395435 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:18.395390 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:23.125074 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:23.125024 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:27.136039 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:27.136005 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6"] Nov 01 06:43:27.136587 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:27.136210 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" podUID="40b323a1-6851-4cef-b5e8-0b211671eb6c" containerName="maas-api" containerID="cri-o://83ed80c9df46dc5e28cf84959a96f61bbd41137bd87880373842aeedbf85d053" gracePeriod=30 Nov 01 06:43:27.256789 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:27.256705 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:27.442742 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:27.442487 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" Nov 01 06:43:27.550249 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:27.550212 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b323a1-6851-4cef-b5e8-0b211671eb6c-kube-api-access-cwn2m\" (UniqueName: \"kubernetes.io/projected/40b323a1-6851-4cef-b5e8-0b211671eb6c-kube-api-access-cwn2m\") pod \"40b323a1-6851-4cef-b5e8-0b211671eb6c\" (UID: \"40b323a1-6851-4cef-b5e8-0b211671eb6c\") " Nov 01 06:43:27.550454 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:27.550293 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40b323a1-6851-4cef-b5e8-0b211671eb6c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b323a1-6851-4cef-b5e8-0b211671eb6c-maas-api-tls\") pod \"40b323a1-6851-4cef-b5e8-0b211671eb6c\" (UID: \"40b323a1-6851-4cef-b5e8-0b211671eb6c\") " Nov 01 06:43:27.550454 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:27.550345 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40b323a1-6851-4cef-b5e8-0b211671eb6c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b323a1-6851-4cef-b5e8-0b211671eb6c-metrics-certs\") pod \"40b323a1-6851-4cef-b5e8-0b211671eb6c\" (UID: \"40b323a1-6851-4cef-b5e8-0b211671eb6c\") " Nov 01 06:43:27.550454 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:27.550383 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b323a1-6851-4cef-b5e8-0b211671eb6c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b323a1-6851-4cef-b5e8-0b211671eb6c-openshift-service-ca\") pod \"40b323a1-6851-4cef-b5e8-0b211671eb6c\" (UID: \"40b323a1-6851-4cef-b5e8-0b211671eb6c\") " Nov 01 06:43:27.551073 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:27.551042 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b323a1-6851-4cef-b5e8-0b211671eb6c-openshift-service-ca" pod "40b323a1-6851-4cef-b5e8-0b211671eb6c" (UID: "40b323a1-6851-4cef-b5e8-0b211671eb6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:43:27.553268 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:27.553230 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b323a1-6851-4cef-b5e8-0b211671eb6c-metrics-certs" pod "40b323a1-6851-4cef-b5e8-0b211671eb6c" (UID: "40b323a1-6851-4cef-b5e8-0b211671eb6c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:43:27.553795 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:27.553773 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b323a1-6851-4cef-b5e8-0b211671eb6c-kube-api-access-cwn2m" pod "40b323a1-6851-4cef-b5e8-0b211671eb6c" (UID: "40b323a1-6851-4cef-b5e8-0b211671eb6c"). InnerVolumeSpecName "kube-api-access-cwn2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:43:27.556245 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:27.556220 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b323a1-6851-4cef-b5e8-0b211671eb6c-maas-api-tls" pod "40b323a1-6851-4cef-b5e8-0b211671eb6c" (UID: "40b323a1-6851-4cef-b5e8-0b211671eb6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:43:27.652302 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:27.652252 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwn2m\" (UniqueName: \"kubernetes.io/projected/40b323a1-6851-4cef-b5e8-0b211671eb6c-kube-api-access-cwn2m\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:27.652302 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:27.652297 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b323a1-6851-4cef-b5e8-0b211671eb6c-maas-api-tls\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:27.652302 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:27.652308 4385 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b323a1-6851-4cef-b5e8-0b211671eb6c-metrics-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:27.652528 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:27.652317 4385 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b323a1-6851-4cef-b5e8-0b211671eb6c-openshift-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:28.243282 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:28.243247 4385 generic.go:356] "Generic (PLEG): container finished" podID="40b323a1-6851-4cef-b5e8-0b211671eb6c" containerID="83ed80c9df46dc5e28cf84959a96f61bbd41137bd87880373842aeedbf85d053" exitCode=0 Nov 01 06:43:28.243777 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:28.243304 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" Nov 01 06:43:28.243777 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:28.243341 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" event={"ID":"40b323a1-6851-4cef-b5e8-0b211671eb6c","Type":"ContainerDied","Data":"83ed80c9df46dc5e28cf84959a96f61bbd41137bd87880373842aeedbf85d053"} Nov 01 06:43:28.243777 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:28.243396 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6" event={"ID":"40b323a1-6851-4cef-b5e8-0b211671eb6c","Type":"ContainerDied","Data":"1d41ebcfb0dc638ff40441d479d4c97eb408eb9dc977d3286ac6e19b3551b8eb"} Nov 01 06:43:28.243777 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:28.243423 4385 scope.go:122] "RemoveContainer" containerID="83ed80c9df46dc5e28cf84959a96f61bbd41137bd87880373842aeedbf85d053" Nov 01 06:43:28.257452 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:28.257370 4385 scope.go:122] "RemoveContainer" containerID="83ed80c9df46dc5e28cf84959a96f61bbd41137bd87880373842aeedbf85d053" Nov 01 06:43:28.258102 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:43:28.258068 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ed80c9df46dc5e28cf84959a96f61bbd41137bd87880373842aeedbf85d053\": container with ID starting with 83ed80c9df46dc5e28cf84959a96f61bbd41137bd87880373842aeedbf85d053 not found: ID does not exist" containerID="83ed80c9df46dc5e28cf84959a96f61bbd41137bd87880373842aeedbf85d053" Nov 01 06:43:28.258207 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:28.258109 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ed80c9df46dc5e28cf84959a96f61bbd41137bd87880373842aeedbf85d053"} err="failed to get container status \"83ed80c9df46dc5e28cf84959a96f61bbd41137bd87880373842aeedbf85d053\": rpc error: code = NotFound desc = could not find container \"83ed80c9df46dc5e28cf84959a96f61bbd41137bd87880373842aeedbf85d053\": container with ID starting with 83ed80c9df46dc5e28cf84959a96f61bbd41137bd87880373842aeedbf85d053 not found: ID does not exist" Nov 01 06:43:28.407947 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:28.407912 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6"] Nov 01 06:43:28.457244 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:28.457215 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c66acdd-55489b5b45-wwsj6"] Nov 01 06:43:28.801998 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:28.801965 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:29.017113 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:29.017071 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:29.162344 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:29.162310 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:29.950894 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:29.950855 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:30.386428 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:30.386393 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b323a1-6851-4cef-b5e8-0b211671eb6c" path="/var/lib/kubelet/pods/40b323a1-6851-4cef-b5e8-0b211671eb6c/volumes" Nov 01 06:43:30.563611 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:30.563572 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:31.198663 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:31.198631 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:31.596535 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:31.596500 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:32.082585 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:32.082554 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:32.218671 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:32.218634 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:33.221962 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:33.221919 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:33.423518 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:33.423478 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:34.258515 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:34.258474 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:34.770343 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:34.770304 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:34.942612 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:34.942550 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:35.103683 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:35.103647 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:36.779801 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:43:36.779764 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:43:37.937284 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:37.937246 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:38.112509 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:38.112467 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:39.146456 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:39.146417 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:43.865464 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:43.865429 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:45.304974 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:45.304937 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:48.522167 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:48.522133 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:53.233563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:53.233527 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:43:57.927282 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:43:57.927250 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:02.648486 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:02.648443 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:02.967927 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:02.967812 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:06.778863 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:44:06.778809 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:44:07.204777 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:07.204741 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7695-0ff146"] Nov 01 06:44:07.205298 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:07.205279 4385 cpu_manager.go:414] "containerMap: removing container" podUID="40b323a1-6851-4cef-b5e8-0b211671eb6c" containerName="maas-api" Nov 01 06:44:07.205400 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:07.205300 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b323a1-6851-4cef-b5e8-0b211671eb6c" containerName="maas-api" Nov 01 06:44:07.205484 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:07.205470 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b323a1-6851-4cef-b5e8-0b211671eb6c" containerName="maas-api" Nov 01 06:44:07.210000 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:07.209980 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-0ff146" Nov 01 06:44:07.264357 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:07.264330 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-0ff146"] Nov 01 06:44:07.295231 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:07.295201 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8jjm\" (UniqueName: \"kubernetes.io/projected/69b5ee8c-c4f8-4376-9583-13118caa2cd7-kube-api-access-b8jjm\") pod \"test-curl-7695-0ff146\" (UID: \"69b5ee8c-c4f8-4376-9583-13118caa2cd7\") " pod="opendatahub/test-curl-7695-0ff146" Nov 01 06:44:07.298165 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:07.298147 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-557zr\"" Nov 01 06:44:07.323847 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:07.323805 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:07.396750 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:07.396713 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8jjm\" (UniqueName: \"kubernetes.io/projected/69b5ee8c-c4f8-4376-9583-13118caa2cd7-kube-api-access-b8jjm\") pod \"test-curl-7695-0ff146\" (UID: \"69b5ee8c-c4f8-4376-9583-13118caa2cd7\") " pod="opendatahub/test-curl-7695-0ff146" Nov 01 06:44:07.453213 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:07.453184 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8jjm\" (UniqueName: \"kubernetes.io/projected/69b5ee8c-c4f8-4376-9583-13118caa2cd7-kube-api-access-b8jjm\") pod \"test-curl-7695-0ff146\" (UID: \"69b5ee8c-c4f8-4376-9583-13118caa2cd7\") " pod="opendatahub/test-curl-7695-0ff146" Nov 01 06:44:07.522121 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:07.522041 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-0ff146" Nov 01 06:44:07.906104 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:07.906075 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-0ff146"] Nov 01 06:44:07.906536 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:44:07.906217 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69b5ee8c_c4f8_4376_9583_13118caa2cd7.slice/crio-6289912eba97737f7c76b09a9268e1ef67f588be8a67286910b06e65c60a8851 WatchSource:0}: Error finding container 6289912eba97737f7c76b09a9268e1ef67f588be8a67286910b06e65c60a8851: Status 404 returned error can't find the container with id 6289912eba97737f7c76b09a9268e1ef67f588be8a67286910b06e65c60a8851 Nov 01 06:44:08.537466 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:08.537432 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-0ff146" event={"ID":"69b5ee8c-c4f8-4376-9583-13118caa2cd7","Type":"ContainerStarted","Data":"6289912eba97737f7c76b09a9268e1ef67f588be8a67286910b06e65c60a8851"} Nov 01 06:44:08.818635 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:08.818548 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:09.555150 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:09.555101 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-0ff146" event={"ID":"69b5ee8c-c4f8-4376-9583-13118caa2cd7","Type":"ContainerStarted","Data":"c9e44587533d9fd77a941d68903c05566839abf6febf6dca529bb974605819b5"} Nov 01 06:44:09.667760 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:09.667710 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7695-0ff146" podStartSLOduration=1.5525620500000001 podStartE2EDuration="2.667696318s" totalImagesPullingTime="1.115134268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:44:09.665378898 +0000 UTC m=+2059.429178757" watchObservedRunningTime="2026-11-01 06:44:09.667696318 +0000 UTC m=+2059.431496174" Nov 01 06:44:11.085981 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:11.085948 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7695-0ff146"] Nov 01 06:44:11.091882 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:11.091850 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7695-0ff146"] Nov 01 06:44:11.580194 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:11.580155 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7695-0ff146" podUID="69b5ee8c-c4f8-4376-9583-13118caa2cd7" containerName="test-curl-7695-0ff146" containerID="cri-o://c9e44587533d9fd77a941d68903c05566839abf6febf6dca529bb974605819b5" gracePeriod=2 Nov 01 06:44:11.631135 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:44:11.631101 4385 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7695-0ff146\" is forbidden: User \"system:node:ip-10-0-140-68.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-68.us-west-1.compute.internal' and this object" podUID="69b5ee8c-c4f8-4376-9583-13118caa2cd7" pod="opendatahub/test-curl-7695-0ff146" Nov 01 06:44:12.034724 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:12.034647 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:13.832401 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:13.832377 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-0ff146" Nov 01 06:44:13.970257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:13.970175 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69b5ee8c-c4f8-4376-9583-13118caa2cd7-kube-api-access-b8jjm\" (UniqueName: \"kubernetes.io/projected/69b5ee8c-c4f8-4376-9583-13118caa2cd7-kube-api-access-b8jjm\") pod \"69b5ee8c-c4f8-4376-9583-13118caa2cd7\" (UID: \"69b5ee8c-c4f8-4376-9583-13118caa2cd7\") " Nov 01 06:44:13.972385 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:13.972356 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b5ee8c-c4f8-4376-9583-13118caa2cd7-kube-api-access-b8jjm" pod "69b5ee8c-c4f8-4376-9583-13118caa2cd7" (UID: "69b5ee8c-c4f8-4376-9583-13118caa2cd7"). InnerVolumeSpecName "kube-api-access-b8jjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:44:14.071893 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:14.071807 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8jjm\" (UniqueName: \"kubernetes.io/projected/69b5ee8c-c4f8-4376-9583-13118caa2cd7-kube-api-access-b8jjm\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:44:14.384673 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:14.384635 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69b5ee8c-c4f8-4376-9583-13118caa2cd7" path="/var/lib/kubelet/pods/69b5ee8c-c4f8-4376-9583-13118caa2cd7/volumes" Nov 01 06:44:14.606210 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:14.606169 4385 generic.go:356] "Generic (PLEG): container finished" podID="69b5ee8c-c4f8-4376-9583-13118caa2cd7" containerID="c9e44587533d9fd77a941d68903c05566839abf6febf6dca529bb974605819b5" exitCode=137 Nov 01 06:44:14.606569 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:14.606553 4385 scope.go:122] "RemoveContainer" containerID="c9e44587533d9fd77a941d68903c05566839abf6febf6dca529bb974605819b5" Nov 01 06:44:14.607395 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:14.606659 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-0ff146" Nov 01 06:44:14.623121 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:14.623091 4385 scope.go:122] "RemoveContainer" containerID="c9e44587533d9fd77a941d68903c05566839abf6febf6dca529bb974605819b5" Nov 01 06:44:14.623485 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:44:14.623464 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e44587533d9fd77a941d68903c05566839abf6febf6dca529bb974605819b5\": container with ID starting with c9e44587533d9fd77a941d68903c05566839abf6febf6dca529bb974605819b5 not found: ID does not exist" containerID="c9e44587533d9fd77a941d68903c05566839abf6febf6dca529bb974605819b5" Nov 01 06:44:14.623555 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:14.623500 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e44587533d9fd77a941d68903c05566839abf6febf6dca529bb974605819b5"} err="failed to get container status \"c9e44587533d9fd77a941d68903c05566839abf6febf6dca529bb974605819b5\": rpc error: code = NotFound desc = could not find container \"c9e44587533d9fd77a941d68903c05566839abf6febf6dca529bb974605819b5\": container with ID starting with c9e44587533d9fd77a941d68903c05566839abf6febf6dca529bb974605819b5 not found: ID does not exist" Nov 01 06:44:16.739285 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:16.739252 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:21.469484 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:21.469449 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr"] Nov 01 06:44:21.470241 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:21.470215 4385 cpu_manager.go:414] "containerMap: removing container" podUID="69b5ee8c-c4f8-4376-9583-13118caa2cd7" containerName="test-curl-7695-0ff146" Nov 01 06:44:21.470416 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:21.470400 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69b5ee8c-c4f8-4376-9583-13118caa2cd7" containerName="test-curl-7695-0ff146" Nov 01 06:44:21.470707 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:21.470687 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="69b5ee8c-c4f8-4376-9583-13118caa2cd7" containerName="test-curl-7695-0ff146" Nov 01 06:44:21.476180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:21.476159 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" Nov 01 06:44:21.543165 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:21.543126 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr"] Nov 01 06:44:21.580694 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:21.580032 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-030936cd-serving-cert\"" Nov 01 06:44:21.664702 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:21.664668 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e107c800-1601-48ae-b4ea-8606ec6c1b67-metrics-certs\") pod \"maas-api-e2e-derive-030936cd-7f57589fff-dlhrr\" (UID: \"e107c800-1601-48ae-b4ea-8606ec6c1b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" Nov 01 06:44:21.664895 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:21.664719 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e107c800-1601-48ae-b4ea-8606ec6c1b67-openshift-service-ca\") pod \"maas-api-e2e-derive-030936cd-7f57589fff-dlhrr\" (UID: \"e107c800-1601-48ae-b4ea-8606ec6c1b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" Nov 01 06:44:21.664895 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:21.664799 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e107c800-1601-48ae-b4ea-8606ec6c1b67-maas-api-tls\") pod \"maas-api-e2e-derive-030936cd-7f57589fff-dlhrr\" (UID: \"e107c800-1601-48ae-b4ea-8606ec6c1b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" Nov 01 06:44:21.665150 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:21.665132 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qc99\" (UniqueName: \"kubernetes.io/projected/e107c800-1601-48ae-b4ea-8606ec6c1b67-kube-api-access-2qc99\") pod \"maas-api-e2e-derive-030936cd-7f57589fff-dlhrr\" (UID: \"e107c800-1601-48ae-b4ea-8606ec6c1b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" Nov 01 06:44:21.703224 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:21.703192 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:21.766545 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:21.766449 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e107c800-1601-48ae-b4ea-8606ec6c1b67-metrics-certs\") pod \"maas-api-e2e-derive-030936cd-7f57589fff-dlhrr\" (UID: \"e107c800-1601-48ae-b4ea-8606ec6c1b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" Nov 01 06:44:21.766545 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:21.766523 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e107c800-1601-48ae-b4ea-8606ec6c1b67-openshift-service-ca\") pod \"maas-api-e2e-derive-030936cd-7f57589fff-dlhrr\" (UID: \"e107c800-1601-48ae-b4ea-8606ec6c1b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" Nov 01 06:44:21.766788 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:21.766580 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e107c800-1601-48ae-b4ea-8606ec6c1b67-maas-api-tls\") pod \"maas-api-e2e-derive-030936cd-7f57589fff-dlhrr\" (UID: \"e107c800-1601-48ae-b4ea-8606ec6c1b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" Nov 01 06:44:21.766788 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:21.766675 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qc99\" (UniqueName: \"kubernetes.io/projected/e107c800-1601-48ae-b4ea-8606ec6c1b67-kube-api-access-2qc99\") pod \"maas-api-e2e-derive-030936cd-7f57589fff-dlhrr\" (UID: \"e107c800-1601-48ae-b4ea-8606ec6c1b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" Nov 01 06:44:21.767595 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:21.767571 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e107c800-1601-48ae-b4ea-8606ec6c1b67-openshift-service-ca\") pod \"maas-api-e2e-derive-030936cd-7f57589fff-dlhrr\" (UID: \"e107c800-1601-48ae-b4ea-8606ec6c1b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" Nov 01 06:44:21.769960 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:21.769924 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e107c800-1601-48ae-b4ea-8606ec6c1b67-metrics-certs\") pod \"maas-api-e2e-derive-030936cd-7f57589fff-dlhrr\" (UID: \"e107c800-1601-48ae-b4ea-8606ec6c1b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" Nov 01 06:44:21.772349 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:21.772319 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e107c800-1601-48ae-b4ea-8606ec6c1b67-maas-api-tls\") pod \"maas-api-e2e-derive-030936cd-7f57589fff-dlhrr\" (UID: \"e107c800-1601-48ae-b4ea-8606ec6c1b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" Nov 01 06:44:21.824860 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:21.824800 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qc99\" (UniqueName: \"kubernetes.io/projected/e107c800-1601-48ae-b4ea-8606ec6c1b67-kube-api-access-2qc99\") pod \"maas-api-e2e-derive-030936cd-7f57589fff-dlhrr\" (UID: \"e107c800-1601-48ae-b4ea-8606ec6c1b67\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" Nov 01 06:44:21.857250 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:21.857217 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:22.007784 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:22.007744 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:22.107464 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:22.107432 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" Nov 01 06:44:22.488405 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:22.488383 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr"] Nov 01 06:44:22.489249 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:44:22.489225 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode107c800_1601_48ae_b4ea_8606ec6c1b67.slice/crio-8345f043824919e4b9c45e73614826e2d9ef1bbe1fd7282a9962efa619d74ee3 WatchSource:0}: Error finding container 8345f043824919e4b9c45e73614826e2d9ef1bbe1fd7282a9962efa619d74ee3: Status 404 returned error can't find the container with id 8345f043824919e4b9c45e73614826e2d9ef1bbe1fd7282a9962efa619d74ee3 Nov 01 06:44:22.677221 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:22.677190 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" event={"ID":"e107c800-1601-48ae-b4ea-8606ec6c1b67","Type":"ContainerStarted","Data":"8345f043824919e4b9c45e73614826e2d9ef1bbe1fd7282a9962efa619d74ee3"} Nov 01 06:44:23.685586 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:23.685499 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" event={"ID":"e107c800-1601-48ae-b4ea-8606ec6c1b67","Type":"ContainerStarted","Data":"529f23ab0cd33d18cb8f949a463672077ebab8fc871c85279f41640ebddae899"} Nov 01 06:44:23.685586 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:23.685565 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" Nov 01 06:44:23.801658 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:23.801598 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" podStartSLOduration=1.904940201 podStartE2EDuration="2.801583636s" totalImagesPullingTime="896.643435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:44:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:44:23.798363887 +0000 UTC m=+2073.562163743" watchObservedRunningTime="2026-11-01 06:44:23.801583636 +0000 UTC m=+2073.565383493" Nov 01 06:44:25.219197 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:25.219159 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:26.530072 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:26.530042 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:29.701620 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:29.701584 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" Nov 01 06:44:31.234204 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:31.234163 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:35.814804 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:35.814767 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:35.975124 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:35.975089 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:36.138279 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:36.138238 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:36.781643 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:44:36.781582 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:44:40.768236 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:40.768204 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:45.381773 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:45.381740 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:45.700638 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:45.700540 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:50.145375 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:50.145338 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:54.805799 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:54.805760 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:56.404151 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:56.404110 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:56.819997 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:56.819889 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:57.390618 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:57.390456 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:57.598883 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:57.598853 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:44:59.501124 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:44:59.501085 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:00.167788 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.167756 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc"] Nov 01 06:45:00.172909 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.172886 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" Nov 01 06:45:00.190583 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.190559 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm"] Nov 01 06:45:00.194511 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.194489 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz"] Nov 01 06:45:00.194629 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.194596 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" Nov 01 06:45:00.198151 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.198133 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz" Nov 01 06:45:00.217878 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.217848 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9"] Nov 01 06:45:00.221603 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.221587 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9" Nov 01 06:45:00.230524 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.230496 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cce3484-b9d3-4d52-a06d-1f407c3a96c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891925-p68mm\" (UID: \"8cce3484-b9d3-4d52-a06d-1f407c3a96c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" Nov 01 06:45:00.230654 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.230559 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5450017c-fdb1-448a-a82d-593f799789c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9\" (UID: \"5450017c-fdb1-448a-a82d-593f799789c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9" Nov 01 06:45:00.230706 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.230650 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhkft\" (UniqueName: \"kubernetes.io/projected/73f8815d-000f-4134-9169-6b686f776e58-kube-api-access-jhkft\") pod \"maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz\" (UID: \"73f8815d-000f-4134-9169-6b686f776e58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz" Nov 01 06:45:00.230751 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.230705 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924d1205-076c-4787-a9ca-8291af9115a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc\" (UID: \"924d1205-076c-4787-a9ca-8291af9115a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" Nov 01 06:45:00.230847 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.230806 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f8815d-000f-4134-9169-6b686f776e58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz\" (UID: \"73f8815d-000f-4134-9169-6b686f776e58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz" Nov 01 06:45:00.230956 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.230937 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czsf4\" (UniqueName: \"kubernetes.io/projected/8cce3484-b9d3-4d52-a06d-1f407c3a96c4-kube-api-access-czsf4\") pod \"maas-api-key-cleanup-29891925-p68mm\" (UID: \"8cce3484-b9d3-4d52-a06d-1f407c3a96c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" Nov 01 06:45:00.231039 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.231022 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bc6n\" (UniqueName: \"kubernetes.io/projected/5450017c-fdb1-448a-a82d-593f799789c4-kube-api-access-7bc6n\") pod \"maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9\" (UID: \"5450017c-fdb1-448a-a82d-593f799789c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9" Nov 01 06:45:00.231121 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.231099 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttbz2\" (UniqueName: \"kubernetes.io/projected/924d1205-076c-4787-a9ca-8291af9115a4-kube-api-access-ttbz2\") pod \"maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc\" (UID: \"924d1205-076c-4787-a9ca-8291af9115a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" Nov 01 06:45:00.243857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.237598 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\"" Nov 01 06:45:00.271646 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.271607 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc"] Nov 01 06:45:00.277006 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.276961 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm"] Nov 01 06:45:00.283096 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.283047 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz"] Nov 01 06:45:00.291134 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.291104 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9"] Nov 01 06:45:00.332040 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.332010 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cce3484-b9d3-4d52-a06d-1f407c3a96c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891925-p68mm\" (UID: \"8cce3484-b9d3-4d52-a06d-1f407c3a96c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" Nov 01 06:45:00.332250 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.332058 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5450017c-fdb1-448a-a82d-593f799789c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9\" (UID: \"5450017c-fdb1-448a-a82d-593f799789c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9" Nov 01 06:45:00.332250 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.332094 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhkft\" (UniqueName: \"kubernetes.io/projected/73f8815d-000f-4134-9169-6b686f776e58-kube-api-access-jhkft\") pod \"maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz\" (UID: \"73f8815d-000f-4134-9169-6b686f776e58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz" Nov 01 06:45:00.332250 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.332135 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924d1205-076c-4787-a9ca-8291af9115a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc\" (UID: \"924d1205-076c-4787-a9ca-8291af9115a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" Nov 01 06:45:00.332250 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.332160 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f8815d-000f-4134-9169-6b686f776e58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz\" (UID: \"73f8815d-000f-4134-9169-6b686f776e58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz" Nov 01 06:45:00.332250 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.332200 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czsf4\" (UniqueName: \"kubernetes.io/projected/8cce3484-b9d3-4d52-a06d-1f407c3a96c4-kube-api-access-czsf4\") pod \"maas-api-key-cleanup-29891925-p68mm\" (UID: \"8cce3484-b9d3-4d52-a06d-1f407c3a96c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" Nov 01 06:45:00.332527 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.332265 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bc6n\" (UniqueName: \"kubernetes.io/projected/5450017c-fdb1-448a-a82d-593f799789c4-kube-api-access-7bc6n\") pod \"maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9\" (UID: \"5450017c-fdb1-448a-a82d-593f799789c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9" Nov 01 06:45:00.332527 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.332302 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttbz2\" (UniqueName: \"kubernetes.io/projected/924d1205-076c-4787-a9ca-8291af9115a4-kube-api-access-ttbz2\") pod \"maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc\" (UID: \"924d1205-076c-4787-a9ca-8291af9115a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" Nov 01 06:45:00.332820 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.332798 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cce3484-b9d3-4d52-a06d-1f407c3a96c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891925-p68mm\" (UID: \"8cce3484-b9d3-4d52-a06d-1f407c3a96c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" Nov 01 06:45:00.332994 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.332975 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5450017c-fdb1-448a-a82d-593f799789c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9\" (UID: \"5450017c-fdb1-448a-a82d-593f799789c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9" Nov 01 06:45:00.333032 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.332995 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924d1205-076c-4787-a9ca-8291af9115a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc\" (UID: \"924d1205-076c-4787-a9ca-8291af9115a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" Nov 01 06:45:00.333177 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.333163 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f8815d-000f-4134-9169-6b686f776e58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz\" (UID: \"73f8815d-000f-4134-9169-6b686f776e58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz" Nov 01 06:45:00.389506 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.389477 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bc6n\" (UniqueName: \"kubernetes.io/projected/5450017c-fdb1-448a-a82d-593f799789c4-kube-api-access-7bc6n\") pod \"maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9\" (UID: \"5450017c-fdb1-448a-a82d-593f799789c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9" Nov 01 06:45:00.389686 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.389630 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czsf4\" (UniqueName: \"kubernetes.io/projected/8cce3484-b9d3-4d52-a06d-1f407c3a96c4-kube-api-access-czsf4\") pod \"maas-api-key-cleanup-29891925-p68mm\" (UID: \"8cce3484-b9d3-4d52-a06d-1f407c3a96c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" Nov 01 06:45:00.389779 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.389740 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhkft\" (UniqueName: \"kubernetes.io/projected/73f8815d-000f-4134-9169-6b686f776e58-kube-api-access-jhkft\") pod \"maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz\" (UID: \"73f8815d-000f-4134-9169-6b686f776e58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz" Nov 01 06:45:00.390002 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.389982 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttbz2\" (UniqueName: \"kubernetes.io/projected/924d1205-076c-4787-a9ca-8291af9115a4-kube-api-access-ttbz2\") pod \"maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc\" (UID: \"924d1205-076c-4787-a9ca-8291af9115a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" Nov 01 06:45:00.486391 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.486313 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" Nov 01 06:45:00.511185 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.511153 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" Nov 01 06:45:00.516385 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.516364 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz" Nov 01 06:45:00.536861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.536815 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9" Nov 01 06:45:00.869789 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.869760 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc"] Nov 01 06:45:00.870476 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:45:00.870452 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924d1205_076c_4787_a9ca_8291af9115a4.slice/crio-39275244930aab55a8db536daca7b481e07b2142bfbacb05f73db35f0d464071 WatchSource:0}: Error finding container 39275244930aab55a8db536daca7b481e07b2142bfbacb05f73db35f0d464071: Status 404 returned error can't find the container with id 39275244930aab55a8db536daca7b481e07b2142bfbacb05f73db35f0d464071 Nov 01 06:45:00.919616 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.919583 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm"] Nov 01 06:45:00.920887 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:45:00.920848 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cce3484_b9d3_4d52_a06d_1f407c3a96c4.slice/crio-7c4fcda9dafaabaa875ca00dc76e05cc4a0d57277a76a38fae8d299edf80bf40 WatchSource:0}: Error finding container 7c4fcda9dafaabaa875ca00dc76e05cc4a0d57277a76a38fae8d299edf80bf40: Status 404 returned error can't find the container with id 7c4fcda9dafaabaa875ca00dc76e05cc4a0d57277a76a38fae8d299edf80bf40 Nov 01 06:45:00.924895 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.924863 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz"] Nov 01 06:45:00.966328 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.966289 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz" event={"ID":"73f8815d-000f-4134-9169-6b686f776e58","Type":"ContainerStarted","Data":"8dfb1493bedd3157a2d360b227f00787e84ddaa23bd57b6fe648d64a3a5a9fe7"} Nov 01 06:45:00.967815 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.967792 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" event={"ID":"924d1205-076c-4787-a9ca-8291af9115a4","Type":"ContainerStarted","Data":"39275244930aab55a8db536daca7b481e07b2142bfbacb05f73db35f0d464071"} Nov 01 06:45:00.969547 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.969522 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" event={"ID":"8cce3484-b9d3-4d52-a06d-1f407c3a96c4","Type":"ContainerStarted","Data":"7c4fcda9dafaabaa875ca00dc76e05cc4a0d57277a76a38fae8d299edf80bf40"} Nov 01 06:45:00.970249 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:00.970230 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9"] Nov 01 06:45:00.972961 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:45:00.972935 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5450017c_fdb1_448a_a82d_593f799789c4.slice/crio-dd9f459b5843dc678b3fde7ff0071b68e28ffa6021ca4d6105cbee68ef38470f WatchSource:0}: Error finding container dd9f459b5843dc678b3fde7ff0071b68e28ffa6021ca4d6105cbee68ef38470f: Status 404 returned error can't find the container with id dd9f459b5843dc678b3fde7ff0071b68e28ffa6021ca4d6105cbee68ef38470f Nov 01 06:45:01.055268 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:01.055208 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:01.218661 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:01.218625 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:01.364305 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:01.364272 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:01.983914 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:01.983883 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" event={"ID":"8cce3484-b9d3-4d52-a06d-1f407c3a96c4","Type":"ContainerStarted","Data":"9408eea47fec4ea9eb5c85d28d16d3a5dd7962341a72dfa729bd19a08c04540e"} Nov 01 06:45:01.984292 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:01.984257 4385 scope.go:122] "RemoveContainer" containerID="9408eea47fec4ea9eb5c85d28d16d3a5dd7962341a72dfa729bd19a08c04540e" Nov 01 06:45:01.985854 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:01.985807 4385 generic.go:356] "Generic (PLEG): container finished" podID="73f8815d-000f-4134-9169-6b686f776e58" containerID="02bd8d65c323f2b5d7a4cebe8f63ec9607ddd6bc567e033f641b094416929116" exitCode=60 Nov 01 06:45:01.985960 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:01.985946 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz" event={"ID":"73f8815d-000f-4134-9169-6b686f776e58","Type":"ContainerDied","Data":"02bd8d65c323f2b5d7a4cebe8f63ec9607ddd6bc567e033f641b094416929116"} Nov 01 06:45:01.986144 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:01.986127 4385 scope.go:122] "RemoveContainer" containerID="02bd8d65c323f2b5d7a4cebe8f63ec9607ddd6bc567e033f641b094416929116" Nov 01 06:45:01.987876 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:01.987771 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9" event={"ID":"5450017c-fdb1-448a-a82d-593f799789c4","Type":"ContainerStarted","Data":"4cdf31692a0453abbff0b43bcad791cc3e6f27be8c0513c1804a61d7f61c39fc"} Nov 01 06:45:01.987876 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:01.987818 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9" event={"ID":"5450017c-fdb1-448a-a82d-593f799789c4","Type":"ContainerStarted","Data":"dd9f459b5843dc678b3fde7ff0071b68e28ffa6021ca4d6105cbee68ef38470f"} Nov 01 06:45:01.989755 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:01.989733 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" event={"ID":"924d1205-076c-4787-a9ca-8291af9115a4","Type":"ContainerStarted","Data":"2e086dfcd144b585f46c9a0aa00d48c7228601b07832125904d9d7b1f59d4393"} Nov 01 06:45:01.990009 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:01.989996 4385 scope.go:122] "RemoveContainer" containerID="2e086dfcd144b585f46c9a0aa00d48c7228601b07832125904d9d7b1f59d4393" Nov 01 06:45:02.318714 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:02.318658 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9" podStartSLOduration=2.318640666 podStartE2EDuration="2.318640666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:02.317802402 +0000 UTC m=+2112.081602259" watchObservedRunningTime="2026-11-01 06:45:02.318640666 +0000 UTC m=+2112.082440525" Nov 01 06:45:02.999486 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:02.999452 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz" event={"ID":"73f8815d-000f-4134-9169-6b686f776e58","Type":"ContainerStarted","Data":"6d59d03f448cba0b2f9da0accc1b7bb99ba868451904056f435821cae90634c8"} Nov 01 06:45:03.001398 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:03.001369 4385 generic.go:356] "Generic (PLEG): container finished" podID="5450017c-fdb1-448a-a82d-593f799789c4" containerID="4cdf31692a0453abbff0b43bcad791cc3e6f27be8c0513c1804a61d7f61c39fc" exitCode=60 Nov 01 06:45:03.001532 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:03.001417 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9" event={"ID":"5450017c-fdb1-448a-a82d-593f799789c4","Type":"ContainerDied","Data":"4cdf31692a0453abbff0b43bcad791cc3e6f27be8c0513c1804a61d7f61c39fc"} Nov 01 06:45:03.001730 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:03.001712 4385 scope.go:122] "RemoveContainer" containerID="4cdf31692a0453abbff0b43bcad791cc3e6f27be8c0513c1804a61d7f61c39fc" Nov 01 06:45:03.003652 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:03.003634 4385 generic.go:356] "Generic (PLEG): container finished" podID="924d1205-076c-4787-a9ca-8291af9115a4" containerID="2e086dfcd144b585f46c9a0aa00d48c7228601b07832125904d9d7b1f59d4393" exitCode=60 Nov 01 06:45:03.003756 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:03.003736 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" event={"ID":"924d1205-076c-4787-a9ca-8291af9115a4","Type":"ContainerDied","Data":"2e086dfcd144b585f46c9a0aa00d48c7228601b07832125904d9d7b1f59d4393"} Nov 01 06:45:03.003863 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:03.003794 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" event={"ID":"924d1205-076c-4787-a9ca-8291af9115a4","Type":"ContainerStarted","Data":"ac0d0b7d771dc59933dc50847f0e1074f4572f713ba3251baf48b55680eeef03"} Nov 01 06:45:03.005941 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:03.005912 4385 generic.go:356] "Generic (PLEG): container finished" podID="8cce3484-b9d3-4d52-a06d-1f407c3a96c4" containerID="9408eea47fec4ea9eb5c85d28d16d3a5dd7962341a72dfa729bd19a08c04540e" exitCode=60 Nov 01 06:45:03.006039 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:03.005985 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" event={"ID":"8cce3484-b9d3-4d52-a06d-1f407c3a96c4","Type":"ContainerDied","Data":"9408eea47fec4ea9eb5c85d28d16d3a5dd7962341a72dfa729bd19a08c04540e"} Nov 01 06:45:03.006039 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:03.006031 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" event={"ID":"8cce3484-b9d3-4d52-a06d-1f407c3a96c4","Type":"ContainerStarted","Data":"19222a96f8cc21d50b29f917a29b0f1a0524d206f440fc4e2b4558ef329f09cc"} Nov 01 06:45:03.113429 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:03.113370 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz" podStartSLOduration=3.113356819 podStartE2EDuration="3.113356819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:03.110874996 +0000 UTC m=+2112.874674855" watchObservedRunningTime="2026-11-01 06:45:03.113356819 +0000 UTC m=+2112.877156676" Nov 01 06:45:03.229779 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:03.229722 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" podStartSLOduration=3.229709386 podStartE2EDuration="3.229709386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:03.226501524 +0000 UTC m=+2112.990301383" watchObservedRunningTime="2026-11-01 06:45:03.229709386 +0000 UTC m=+2112.993509242" Nov 01 06:45:03.338148 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:03.338085 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" podStartSLOduration=3.338065684 podStartE2EDuration="3.338065684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:03.337468323 +0000 UTC m=+2113.101268182" watchObservedRunningTime="2026-11-01 06:45:03.338065684 +0000 UTC m=+2113.101865542" Nov 01 06:45:04.015133 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:04.015100 4385 generic.go:356] "Generic (PLEG): container finished" podID="8cce3484-b9d3-4d52-a06d-1f407c3a96c4" containerID="19222a96f8cc21d50b29f917a29b0f1a0524d206f440fc4e2b4558ef329f09cc" exitCode=60 Nov 01 06:45:04.015572 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:04.015187 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" event={"ID":"8cce3484-b9d3-4d52-a06d-1f407c3a96c4","Type":"ContainerDied","Data":"19222a96f8cc21d50b29f917a29b0f1a0524d206f440fc4e2b4558ef329f09cc"} Nov 01 06:45:04.015572 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:04.015259 4385 scope.go:122] "RemoveContainer" containerID="9408eea47fec4ea9eb5c85d28d16d3a5dd7962341a72dfa729bd19a08c04540e" Nov 01 06:45:04.015572 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:04.015524 4385 scope.go:122] "RemoveContainer" containerID="19222a96f8cc21d50b29f917a29b0f1a0524d206f440fc4e2b4558ef329f09cc" Nov 01 06:45:04.015891 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:45:04.015870 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891925-p68mm_odh-ai-gateway-infra(8cce3484-b9d3-4d52-a06d-1f407c3a96c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" podUID="8cce3484-b9d3-4d52-a06d-1f407c3a96c4" Nov 01 06:45:04.017616 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:04.017597 4385 generic.go:356] "Generic (PLEG): container finished" podID="73f8815d-000f-4134-9169-6b686f776e58" containerID="6d59d03f448cba0b2f9da0accc1b7bb99ba868451904056f435821cae90634c8" exitCode=60 Nov 01 06:45:04.017687 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:04.017676 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz" event={"ID":"73f8815d-000f-4134-9169-6b686f776e58","Type":"ContainerDied","Data":"6d59d03f448cba0b2f9da0accc1b7bb99ba868451904056f435821cae90634c8"} Nov 01 06:45:04.018044 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:04.018026 4385 scope.go:122] "RemoveContainer" containerID="6d59d03f448cba0b2f9da0accc1b7bb99ba868451904056f435821cae90634c8" Nov 01 06:45:04.018357 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:45:04.018336 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz_odh-ai-gateway-infra(73f8815d-000f-4134-9169-6b686f776e58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz" podUID="73f8815d-000f-4134-9169-6b686f776e58" Nov 01 06:45:04.019917 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:04.019886 4385 generic.go:356] "Generic (PLEG): container finished" podID="5450017c-fdb1-448a-a82d-593f799789c4" containerID="fc2556eeb15050ccdae46161dc6bf70e3c3c870a13d1ab81d5e0172a58f6261e" exitCode=60 Nov 01 06:45:04.020027 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:04.020012 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9" event={"ID":"5450017c-fdb1-448a-a82d-593f799789c4","Type":"ContainerDied","Data":"fc2556eeb15050ccdae46161dc6bf70e3c3c870a13d1ab81d5e0172a58f6261e"} Nov 01 06:45:04.020404 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:04.020384 4385 scope.go:122] "RemoveContainer" containerID="fc2556eeb15050ccdae46161dc6bf70e3c3c870a13d1ab81d5e0172a58f6261e" Nov 01 06:45:04.020715 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:45:04.020689 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9_odh-ai-gateway-infra(5450017c-fdb1-448a-a82d-593f799789c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9" podUID="5450017c-fdb1-448a-a82d-593f799789c4" Nov 01 06:45:04.022141 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:04.022119 4385 generic.go:356] "Generic (PLEG): container finished" podID="924d1205-076c-4787-a9ca-8291af9115a4" containerID="ac0d0b7d771dc59933dc50847f0e1074f4572f713ba3251baf48b55680eeef03" exitCode=60 Nov 01 06:45:04.022228 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:04.022202 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" event={"ID":"924d1205-076c-4787-a9ca-8291af9115a4","Type":"ContainerDied","Data":"ac0d0b7d771dc59933dc50847f0e1074f4572f713ba3251baf48b55680eeef03"} Nov 01 06:45:04.022387 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:04.022373 4385 scope.go:122] "RemoveContainer" containerID="ac0d0b7d771dc59933dc50847f0e1074f4572f713ba3251baf48b55680eeef03" Nov 01 06:45:04.022598 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:45:04.022581 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc_odh-ai-gateway-infra(924d1205-076c-4787-a9ca-8291af9115a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" podUID="924d1205-076c-4787-a9ca-8291af9115a4" Nov 01 06:45:04.038234 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:04.038215 4385 scope.go:122] "RemoveContainer" containerID="02bd8d65c323f2b5d7a4cebe8f63ec9607ddd6bc567e033f641b094416929116" Nov 01 06:45:04.050961 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:04.050939 4385 scope.go:122] "RemoveContainer" containerID="4cdf31692a0453abbff0b43bcad791cc3e6f27be8c0513c1804a61d7f61c39fc" Nov 01 06:45:04.064145 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:04.064127 4385 scope.go:122] "RemoveContainer" containerID="2e086dfcd144b585f46c9a0aa00d48c7228601b07832125904d9d7b1f59d4393" Nov 01 06:45:04.306991 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:04.306896 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:04.454459 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:04.454420 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:04.792193 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:04.792155 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:05.030799 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:05.030756 4385 scope.go:122] "RemoveContainer" containerID="19222a96f8cc21d50b29f917a29b0f1a0524d206f440fc4e2b4558ef329f09cc" Nov 01 06:45:05.031284 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:45:05.031098 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891925-p68mm_odh-ai-gateway-infra(8cce3484-b9d3-4d52-a06d-1f407c3a96c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" podUID="8cce3484-b9d3-4d52-a06d-1f407c3a96c4" Nov 01 06:45:05.032592 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:05.032573 4385 scope.go:122] "RemoveContainer" containerID="6d59d03f448cba0b2f9da0accc1b7bb99ba868451904056f435821cae90634c8" Nov 01 06:45:05.032799 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:45:05.032779 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz_odh-ai-gateway-infra(73f8815d-000f-4134-9169-6b686f776e58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz" podUID="73f8815d-000f-4134-9169-6b686f776e58" Nov 01 06:45:05.034126 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:05.034110 4385 scope.go:122] "RemoveContainer" containerID="fc2556eeb15050ccdae46161dc6bf70e3c3c870a13d1ab81d5e0172a58f6261e" Nov 01 06:45:05.034352 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:45:05.034330 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9_odh-ai-gateway-infra(5450017c-fdb1-448a-a82d-593f799789c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9" podUID="5450017c-fdb1-448a-a82d-593f799789c4" Nov 01 06:45:05.035671 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:05.035655 4385 scope.go:122] "RemoveContainer" containerID="ac0d0b7d771dc59933dc50847f0e1074f4572f713ba3251baf48b55680eeef03" Nov 01 06:45:05.035872 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:45:05.035825 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc_odh-ai-gateway-infra(924d1205-076c-4787-a9ca-8291af9115a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" podUID="924d1205-076c-4787-a9ca-8291af9115a4" Nov 01 06:45:06.779624 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:45:06.779582 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:45:09.718490 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:09.718454 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:13.377483 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:13.377449 4385 scope.go:122] "RemoveContainer" containerID="ac0d0b7d771dc59933dc50847f0e1074f4572f713ba3251baf48b55680eeef03" Nov 01 06:45:14.097720 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:14.097685 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" event={"ID":"924d1205-076c-4787-a9ca-8291af9115a4","Type":"ContainerStarted","Data":"09c8e04386b08625c64df34ec6dc36118b5a7899196f02d5a35fd345cbb4f860"} Nov 01 06:45:14.378344 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:14.378249 4385 scope.go:122] "RemoveContainer" containerID="19222a96f8cc21d50b29f917a29b0f1a0524d206f440fc4e2b4558ef329f09cc" Nov 01 06:45:14.378344 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:14.378315 4385 scope.go:122] "RemoveContainer" containerID="6d59d03f448cba0b2f9da0accc1b7bb99ba868451904056f435821cae90634c8" Nov 01 06:45:14.445606 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:14.445564 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:15.105690 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:15.105656 4385 generic.go:356] "Generic (PLEG): container finished" podID="924d1205-076c-4787-a9ca-8291af9115a4" containerID="09c8e04386b08625c64df34ec6dc36118b5a7899196f02d5a35fd345cbb4f860" exitCode=60 Nov 01 06:45:15.105924 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:15.105730 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" event={"ID":"924d1205-076c-4787-a9ca-8291af9115a4","Type":"ContainerDied","Data":"09c8e04386b08625c64df34ec6dc36118b5a7899196f02d5a35fd345cbb4f860"} Nov 01 06:45:15.105924 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:15.105826 4385 scope.go:122] "RemoveContainer" containerID="ac0d0b7d771dc59933dc50847f0e1074f4572f713ba3251baf48b55680eeef03" Nov 01 06:45:15.106244 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:15.106224 4385 scope.go:122] "RemoveContainer" containerID="09c8e04386b08625c64df34ec6dc36118b5a7899196f02d5a35fd345cbb4f860" Nov 01 06:45:15.106575 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:45:15.106551 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc_odh-ai-gateway-infra(924d1205-076c-4787-a9ca-8291af9115a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" podUID="924d1205-076c-4787-a9ca-8291af9115a4" Nov 01 06:45:15.107895 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:15.107874 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" event={"ID":"8cce3484-b9d3-4d52-a06d-1f407c3a96c4","Type":"ContainerStarted","Data":"b76c1d5fd34e0dc66c852a733bc322e4b05b2196c4942bf1e69424b22408715d"} Nov 01 06:45:15.109737 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:15.109716 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz" event={"ID":"73f8815d-000f-4134-9169-6b686f776e58","Type":"ContainerStarted","Data":"d6729dbb77985fdf63b97116271b0e8c951c3a9b889b29b4c5b28780dffb87f4"} Nov 01 06:45:15.222720 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:15.222689 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc"] Nov 01 06:45:15.377330 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:15.377303 4385 scope.go:122] "RemoveContainer" containerID="fc2556eeb15050ccdae46161dc6bf70e3c3c870a13d1ab81d5e0172a58f6261e" Nov 01 06:45:16.118691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:16.118657 4385 generic.go:356] "Generic (PLEG): container finished" podID="8cce3484-b9d3-4d52-a06d-1f407c3a96c4" containerID="b76c1d5fd34e0dc66c852a733bc322e4b05b2196c4942bf1e69424b22408715d" exitCode=60 Nov 01 06:45:16.119168 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:16.118732 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" event={"ID":"8cce3484-b9d3-4d52-a06d-1f407c3a96c4","Type":"ContainerDied","Data":"b76c1d5fd34e0dc66c852a733bc322e4b05b2196c4942bf1e69424b22408715d"} Nov 01 06:45:16.119168 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:16.118793 4385 scope.go:122] "RemoveContainer" containerID="19222a96f8cc21d50b29f917a29b0f1a0524d206f440fc4e2b4558ef329f09cc" Nov 01 06:45:16.119264 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:16.119168 4385 scope.go:122] "RemoveContainer" containerID="b76c1d5fd34e0dc66c852a733bc322e4b05b2196c4942bf1e69424b22408715d" Nov 01 06:45:16.119494 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:45:16.119474 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891925-p68mm_odh-ai-gateway-infra(8cce3484-b9d3-4d52-a06d-1f407c3a96c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" podUID="8cce3484-b9d3-4d52-a06d-1f407c3a96c4" Nov 01 06:45:16.120751 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:16.120729 4385 generic.go:356] "Generic (PLEG): container finished" podID="73f8815d-000f-4134-9169-6b686f776e58" containerID="d6729dbb77985fdf63b97116271b0e8c951c3a9b889b29b4c5b28780dffb87f4" exitCode=60 Nov 01 06:45:16.120881 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:16.120796 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz" event={"ID":"73f8815d-000f-4134-9169-6b686f776e58","Type":"ContainerDied","Data":"d6729dbb77985fdf63b97116271b0e8c951c3a9b889b29b4c5b28780dffb87f4"} Nov 01 06:45:16.121155 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:16.121137 4385 scope.go:122] "RemoveContainer" containerID="d6729dbb77985fdf63b97116271b0e8c951c3a9b889b29b4c5b28780dffb87f4" Nov 01 06:45:16.121430 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:45:16.121409 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz_odh-ai-gateway-infra(73f8815d-000f-4134-9169-6b686f776e58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz" podUID="73f8815d-000f-4134-9169-6b686f776e58" Nov 01 06:45:16.122601 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:16.122580 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9" event={"ID":"5450017c-fdb1-448a-a82d-593f799789c4","Type":"ContainerStarted","Data":"701f3376cc0dc7642db75c72d0bcc917ad377e8e6fd7114d791f597a4d54b93f"} Nov 01 06:45:16.133093 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:16.133071 4385 scope.go:122] "RemoveContainer" containerID="6d59d03f448cba0b2f9da0accc1b7bb99ba868451904056f435821cae90634c8" Nov 01 06:45:16.344901 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:16.344869 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" Nov 01 06:45:16.357180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:16.357147 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz"] Nov 01 06:45:16.431219 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:16.431130 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924d1205-076c-4787-a9ca-8291af9115a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924d1205-076c-4787-a9ca-8291af9115a4-maas-api-service-ca\") pod \"924d1205-076c-4787-a9ca-8291af9115a4\" (UID: \"924d1205-076c-4787-a9ca-8291af9115a4\") " Nov 01 06:45:16.431219 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:16.431176 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924d1205-076c-4787-a9ca-8291af9115a4-kube-api-access-ttbz2\" (UniqueName: \"kubernetes.io/projected/924d1205-076c-4787-a9ca-8291af9115a4-kube-api-access-ttbz2\") pod \"924d1205-076c-4787-a9ca-8291af9115a4\" (UID: \"924d1205-076c-4787-a9ca-8291af9115a4\") " Nov 01 06:45:16.431524 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:16.431502 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924d1205-076c-4787-a9ca-8291af9115a4-maas-api-service-ca" pod "924d1205-076c-4787-a9ca-8291af9115a4" (UID: "924d1205-076c-4787-a9ca-8291af9115a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:45:16.431708 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:16.431692 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924d1205-076c-4787-a9ca-8291af9115a4-maas-api-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:16.433712 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:16.433684 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924d1205-076c-4787-a9ca-8291af9115a4-kube-api-access-ttbz2" pod "924d1205-076c-4787-a9ca-8291af9115a4" (UID: "924d1205-076c-4787-a9ca-8291af9115a4"). InnerVolumeSpecName "kube-api-access-ttbz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:45:16.466300 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:16.466264 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm"] Nov 01 06:45:16.532692 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:16.532652 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttbz2\" (UniqueName: \"kubernetes.io/projected/924d1205-076c-4787-a9ca-8291af9115a4-kube-api-access-ttbz2\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:17.135979 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.135940 4385 generic.go:356] "Generic (PLEG): container finished" podID="5450017c-fdb1-448a-a82d-593f799789c4" containerID="701f3376cc0dc7642db75c72d0bcc917ad377e8e6fd7114d791f597a4d54b93f" exitCode=60 Nov 01 06:45:17.136420 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.136018 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9" event={"ID":"5450017c-fdb1-448a-a82d-593f799789c4","Type":"ContainerDied","Data":"701f3376cc0dc7642db75c72d0bcc917ad377e8e6fd7114d791f597a4d54b93f"} Nov 01 06:45:17.136420 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.136092 4385 scope.go:122] "RemoveContainer" containerID="fc2556eeb15050ccdae46161dc6bf70e3c3c870a13d1ab81d5e0172a58f6261e" Nov 01 06:45:17.136523 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.136507 4385 scope.go:122] "RemoveContainer" containerID="701f3376cc0dc7642db75c72d0bcc917ad377e8e6fd7114d791f597a4d54b93f" Nov 01 06:45:17.136928 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:45:17.136895 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9_odh-ai-gateway-infra(5450017c-fdb1-448a-a82d-593f799789c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9" podUID="5450017c-fdb1-448a-a82d-593f799789c4" Nov 01 06:45:17.138144 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.138118 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" Nov 01 06:45:17.138316 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.138143 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc" event={"ID":"924d1205-076c-4787-a9ca-8291af9115a4","Type":"ContainerDied","Data":"39275244930aab55a8db536daca7b481e07b2142bfbacb05f73db35f0d464071"} Nov 01 06:45:17.152153 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.152128 4385 scope.go:122] "RemoveContainer" containerID="09c8e04386b08625c64df34ec6dc36118b5a7899196f02d5a35fd345cbb4f860" Nov 01 06:45:17.321241 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.321217 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz" Nov 01 06:45:17.341602 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.341570 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" Nov 01 06:45:17.356495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.356461 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc"] Nov 01 06:45:17.364252 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.364218 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891925-9lrfc"] Nov 01 06:45:17.446462 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.446380 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f8815d-000f-4134-9169-6b686f776e58-kube-api-access-jhkft\" (UniqueName: \"kubernetes.io/projected/73f8815d-000f-4134-9169-6b686f776e58-kube-api-access-jhkft\") pod \"73f8815d-000f-4134-9169-6b686f776e58\" (UID: \"73f8815d-000f-4134-9169-6b686f776e58\") " Nov 01 06:45:17.446612 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.446541 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cce3484-b9d3-4d52-a06d-1f407c3a96c4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cce3484-b9d3-4d52-a06d-1f407c3a96c4-maas-api-service-ca\") pod \"8cce3484-b9d3-4d52-a06d-1f407c3a96c4\" (UID: \"8cce3484-b9d3-4d52-a06d-1f407c3a96c4\") " Nov 01 06:45:17.446663 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.446612 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cce3484-b9d3-4d52-a06d-1f407c3a96c4-kube-api-access-czsf4\" (UniqueName: \"kubernetes.io/projected/8cce3484-b9d3-4d52-a06d-1f407c3a96c4-kube-api-access-czsf4\") pod \"8cce3484-b9d3-4d52-a06d-1f407c3a96c4\" (UID: \"8cce3484-b9d3-4d52-a06d-1f407c3a96c4\") " Nov 01 06:45:17.446663 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.446651 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73f8815d-000f-4134-9169-6b686f776e58-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f8815d-000f-4134-9169-6b686f776e58-maas-api-service-ca\") pod \"73f8815d-000f-4134-9169-6b686f776e58\" (UID: \"73f8815d-000f-4134-9169-6b686f776e58\") " Nov 01 06:45:17.447002 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.446973 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cce3484-b9d3-4d52-a06d-1f407c3a96c4-maas-api-service-ca" pod "8cce3484-b9d3-4d52-a06d-1f407c3a96c4" (UID: "8cce3484-b9d3-4d52-a06d-1f407c3a96c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:45:17.447211 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.447160 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f8815d-000f-4134-9169-6b686f776e58-maas-api-service-ca" pod "73f8815d-000f-4134-9169-6b686f776e58" (UID: "73f8815d-000f-4134-9169-6b686f776e58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:45:17.448852 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.448815 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f8815d-000f-4134-9169-6b686f776e58-kube-api-access-jhkft" pod "73f8815d-000f-4134-9169-6b686f776e58" (UID: "73f8815d-000f-4134-9169-6b686f776e58"). InnerVolumeSpecName "kube-api-access-jhkft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:45:17.448991 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.448971 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cce3484-b9d3-4d52-a06d-1f407c3a96c4-kube-api-access-czsf4" pod "8cce3484-b9d3-4d52-a06d-1f407c3a96c4" (UID: "8cce3484-b9d3-4d52-a06d-1f407c3a96c4"). InnerVolumeSpecName "kube-api-access-czsf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:45:17.491583 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.491550 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9"] Nov 01 06:45:17.547599 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.547566 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cce3484-b9d3-4d52-a06d-1f407c3a96c4-maas-api-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:17.547599 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.547594 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czsf4\" (UniqueName: \"kubernetes.io/projected/8cce3484-b9d3-4d52-a06d-1f407c3a96c4-kube-api-access-czsf4\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:17.547599 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.547607 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f8815d-000f-4134-9169-6b686f776e58-maas-api-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:17.547869 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:17.547616 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhkft\" (UniqueName: \"kubernetes.io/projected/73f8815d-000f-4134-9169-6b686f776e58-kube-api-access-jhkft\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:18.147755 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:18.147722 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz" Nov 01 06:45:18.148267 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:18.147721 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz" event={"ID":"73f8815d-000f-4134-9169-6b686f776e58","Type":"ContainerDied","Data":"8dfb1493bedd3157a2d360b227f00787e84ddaa23bd57b6fe648d64a3a5a9fe7"} Nov 01 06:45:18.148267 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:18.147914 4385 scope.go:122] "RemoveContainer" containerID="d6729dbb77985fdf63b97116271b0e8c951c3a9b889b29b4c5b28780dffb87f4" Nov 01 06:45:18.152374 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:18.152335 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" event={"ID":"8cce3484-b9d3-4d52-a06d-1f407c3a96c4","Type":"ContainerDied","Data":"7c4fcda9dafaabaa875ca00dc76e05cc4a0d57277a76a38fae8d299edf80bf40"} Nov 01 06:45:18.152481 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:18.152374 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm" Nov 01 06:45:18.160207 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:18.160190 4385 scope.go:122] "RemoveContainer" containerID="b76c1d5fd34e0dc66c852a733bc322e4b05b2196c4942bf1e69424b22408715d" Nov 01 06:45:18.300632 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:18.300604 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9" Nov 01 06:45:18.312223 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:18.312189 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm"] Nov 01 06:45:18.315559 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:18.315540 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891925-p68mm"] Nov 01 06:45:18.384101 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:18.384070 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cce3484-b9d3-4d52-a06d-1f407c3a96c4" path="/var/lib/kubelet/pods/8cce3484-b9d3-4d52-a06d-1f407c3a96c4/volumes" Nov 01 06:45:18.384500 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:18.384487 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="924d1205-076c-4787-a9ca-8291af9115a4" path="/var/lib/kubelet/pods/924d1205-076c-4787-a9ca-8291af9115a4/volumes" Nov 01 06:45:18.457553 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:18.457452 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5450017c-fdb1-448a-a82d-593f799789c4-kube-api-access-7bc6n\" (UniqueName: \"kubernetes.io/projected/5450017c-fdb1-448a-a82d-593f799789c4-kube-api-access-7bc6n\") pod \"5450017c-fdb1-448a-a82d-593f799789c4\" (UID: \"5450017c-fdb1-448a-a82d-593f799789c4\") " Nov 01 06:45:18.457553 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:18.457547 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5450017c-fdb1-448a-a82d-593f799789c4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5450017c-fdb1-448a-a82d-593f799789c4-maas-api-service-ca\") pod \"5450017c-fdb1-448a-a82d-593f799789c4\" (UID: \"5450017c-fdb1-448a-a82d-593f799789c4\") " Nov 01 06:45:18.458049 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:18.458019 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5450017c-fdb1-448a-a82d-593f799789c4-maas-api-service-ca" pod "5450017c-fdb1-448a-a82d-593f799789c4" (UID: "5450017c-fdb1-448a-a82d-593f799789c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:45:18.459932 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:18.459902 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5450017c-fdb1-448a-a82d-593f799789c4-kube-api-access-7bc6n" pod "5450017c-fdb1-448a-a82d-593f799789c4" (UID: "5450017c-fdb1-448a-a82d-593f799789c4"). InnerVolumeSpecName "kube-api-access-7bc6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:45:18.538122 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:18.538073 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz"] Nov 01 06:45:18.542298 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:18.542262 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-030936cd-29891925-gr9tz"] Nov 01 06:45:18.558516 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:18.558488 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5450017c-fdb1-448a-a82d-593f799789c4-maas-api-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:18.558516 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:18.558513 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bc6n\" (UniqueName: \"kubernetes.io/projected/5450017c-fdb1-448a-a82d-593f799789c4-kube-api-access-7bc6n\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:19.162752 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:19.162724 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9" Nov 01 06:45:19.162752 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:19.162754 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9" event={"ID":"5450017c-fdb1-448a-a82d-593f799789c4","Type":"ContainerDied","Data":"dd9f459b5843dc678b3fde7ff0071b68e28ffa6021ca4d6105cbee68ef38470f"} Nov 01 06:45:19.163300 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:19.162823 4385 scope.go:122] "RemoveContainer" containerID="701f3376cc0dc7642db75c72d0bcc917ad377e8e6fd7114d791f597a4d54b93f" Nov 01 06:45:19.254986 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:19.254946 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:19.326193 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:19.326065 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9"] Nov 01 06:45:19.329428 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:19.329402 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-190724-29891925-sdth9"] Nov 01 06:45:20.384871 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:20.384820 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5450017c-fdb1-448a-a82d-593f799789c4" path="/var/lib/kubelet/pods/5450017c-fdb1-448a-a82d-593f799789c4/volumes" Nov 01 06:45:20.385262 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:20.385233 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73f8815d-000f-4134-9169-6b686f776e58" path="/var/lib/kubelet/pods/73f8815d-000f-4134-9169-6b686f776e58/volumes" Nov 01 06:45:24.433373 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:24.433341 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:29.563131 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:29.563097 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:33.645908 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:33.645863 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:33.904723 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:33.904631 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:34.057372 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:34.057326 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:34.375629 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:34.375593 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:34.636497 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:34.636409 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:34.864996 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:34.864959 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:36.780151 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:45:36.780106 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:45:38.556530 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:38.556493 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:38.815147 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:38.815059 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:38.972319 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:38.971668 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:39.306786 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:39.306750 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:41.786547 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:41.786512 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:41.983169 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:41.983136 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:44.477629 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:44.477594 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:45.983608 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:45.983568 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:49.333567 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:49.333532 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:53.963353 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:53.963315 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:54.158646 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:54.158611 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:54.309632 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:54.309544 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:45:58.834558 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:45:58.834521 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:03.501253 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:03.501217 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:06.781413 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:46:06.781373 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:46:08.244061 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:08.244020 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:12.242467 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:12.242430 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr"] Nov 01 06:46:12.243013 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:12.242667 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" podUID="e107c800-1601-48ae-b4ea-8606ec6c1b67" containerName="maas-api" containerID="cri-o://529f23ab0cd33d18cb8f949a463672077ebab8fc871c85279f41640ebddae899" gracePeriod=30 Nov 01 06:46:12.363510 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:12.363477 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:12.547712 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:12.547679 4385 generic.go:356] "Generic (PLEG): container finished" podID="e107c800-1601-48ae-b4ea-8606ec6c1b67" containerID="529f23ab0cd33d18cb8f949a463672077ebab8fc871c85279f41640ebddae899" exitCode=0 Nov 01 06:46:12.548125 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:12.548101 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" event={"ID":"e107c800-1601-48ae-b4ea-8606ec6c1b67","Type":"ContainerDied","Data":"529f23ab0cd33d18cb8f949a463672077ebab8fc871c85279f41640ebddae899"} Nov 01 06:46:12.593600 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:12.593278 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" Nov 01 06:46:12.693375 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:12.693332 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e107c800-1601-48ae-b4ea-8606ec6c1b67-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e107c800-1601-48ae-b4ea-8606ec6c1b67-openshift-service-ca\") pod \"e107c800-1601-48ae-b4ea-8606ec6c1b67\" (UID: \"e107c800-1601-48ae-b4ea-8606ec6c1b67\") " Nov 01 06:46:12.693586 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:12.693492 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e107c800-1601-48ae-b4ea-8606ec6c1b67-kube-api-access-2qc99\" (UniqueName: \"kubernetes.io/projected/e107c800-1601-48ae-b4ea-8606ec6c1b67-kube-api-access-2qc99\") pod \"e107c800-1601-48ae-b4ea-8606ec6c1b67\" (UID: \"e107c800-1601-48ae-b4ea-8606ec6c1b67\") " Nov 01 06:46:12.693651 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:12.693623 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e107c800-1601-48ae-b4ea-8606ec6c1b67-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e107c800-1601-48ae-b4ea-8606ec6c1b67-metrics-certs\") pod \"e107c800-1601-48ae-b4ea-8606ec6c1b67\" (UID: \"e107c800-1601-48ae-b4ea-8606ec6c1b67\") " Nov 01 06:46:12.693651 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:12.693634 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e107c800-1601-48ae-b4ea-8606ec6c1b67-openshift-service-ca" pod "e107c800-1601-48ae-b4ea-8606ec6c1b67" (UID: "e107c800-1601-48ae-b4ea-8606ec6c1b67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:46:12.693762 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:12.693662 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e107c800-1601-48ae-b4ea-8606ec6c1b67-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e107c800-1601-48ae-b4ea-8606ec6c1b67-maas-api-tls\") pod \"e107c800-1601-48ae-b4ea-8606ec6c1b67\" (UID: \"e107c800-1601-48ae-b4ea-8606ec6c1b67\") " Nov 01 06:46:12.694107 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:12.694083 4385 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e107c800-1601-48ae-b4ea-8606ec6c1b67-openshift-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:46:12.696056 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:12.696024 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e107c800-1601-48ae-b4ea-8606ec6c1b67-kube-api-access-2qc99" pod "e107c800-1601-48ae-b4ea-8606ec6c1b67" (UID: "e107c800-1601-48ae-b4ea-8606ec6c1b67"). InnerVolumeSpecName "kube-api-access-2qc99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:46:12.696508 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:12.696482 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e107c800-1601-48ae-b4ea-8606ec6c1b67-maas-api-tls" pod "e107c800-1601-48ae-b4ea-8606ec6c1b67" (UID: "e107c800-1601-48ae-b4ea-8606ec6c1b67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:46:12.696625 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:12.696610 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e107c800-1601-48ae-b4ea-8606ec6c1b67-metrics-certs" pod "e107c800-1601-48ae-b4ea-8606ec6c1b67" (UID: "e107c800-1601-48ae-b4ea-8606ec6c1b67"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:46:12.795136 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:12.795100 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qc99\" (UniqueName: \"kubernetes.io/projected/e107c800-1601-48ae-b4ea-8606ec6c1b67-kube-api-access-2qc99\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:46:12.795448 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:12.795431 4385 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e107c800-1601-48ae-b4ea-8606ec6c1b67-metrics-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:46:12.795597 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:12.795584 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e107c800-1601-48ae-b4ea-8606ec6c1b67-maas-api-tls\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:46:13.556496 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:13.556469 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" Nov 01 06:46:13.556970 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:13.556468 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr" event={"ID":"e107c800-1601-48ae-b4ea-8606ec6c1b67","Type":"ContainerDied","Data":"8345f043824919e4b9c45e73614826e2d9ef1bbe1fd7282a9962efa619d74ee3"} Nov 01 06:46:13.556970 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:13.556612 4385 scope.go:122] "RemoveContainer" containerID="529f23ab0cd33d18cb8f949a463672077ebab8fc871c85279f41640ebddae899" Nov 01 06:46:13.720123 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:13.720087 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr"] Nov 01 06:46:13.770057 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:13.770017 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-030936cd-7f57589fff-dlhrr"] Nov 01 06:46:13.907273 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:13.907234 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:14.197597 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:14.197513 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:14.350616 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:14.350587 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:14.385993 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:14.385957 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e107c800-1601-48ae-b4ea-8606ec6c1b67" path="/var/lib/kubelet/pods/e107c800-1601-48ae-b4ea-8606ec6c1b67/volumes" Nov 01 06:46:14.520820 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:14.520732 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:16.381387 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:16.381345 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:16.954856 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:16.954538 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:17.357376 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:17.357332 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:19.247648 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:19.247611 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:21.026746 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:21.026711 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:21.207614 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:21.207575 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:21.358309 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:21.358267 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:24.206901 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:24.206866 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:24.409033 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:24.409001 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:24.565750 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:24.565658 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:28.917220 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:28.917180 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:29.122525 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:29.122494 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:29.275669 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:29.275580 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:33.703864 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:33.703758 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:33.898245 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:33.898213 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:34.049415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:34.049333 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:36.779630 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:46:36.779589 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:46:38.555776 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:38.555741 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:43.316020 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:43.315965 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:46.325620 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:46.325563 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:48.127748 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:48.127709 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:52.819110 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:52.818988 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:57.683120 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:57.683087 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:58.228165 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:58.228136 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:58.575339 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:58.575307 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:59.046734 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:59.046641 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:59.650127 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:59.650092 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:46:59.916545 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:46:59.916469 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:02.926907 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:02.926872 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:03.665928 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:03.665892 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:04.087118 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:04.087074 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:04.291038 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:04.291002 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:04.452406 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:04.452309 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:04.856635 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:04.856602 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:05.894510 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.894479 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh"] Nov 01 06:47:05.894983 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.894970 4385 cpu_manager.go:414] "containerMap: removing container" podUID="924d1205-076c-4787-a9ca-8291af9115a4" containerName="cleanup" Nov 01 06:47:05.895031 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.894984 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924d1205-076c-4787-a9ca-8291af9115a4" containerName="cleanup" Nov 01 06:47:05.895031 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.894995 4385 cpu_manager.go:414] "containerMap: removing container" podUID="924d1205-076c-4787-a9ca-8291af9115a4" containerName="cleanup" Nov 01 06:47:05.895031 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895000 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924d1205-076c-4787-a9ca-8291af9115a4" containerName="cleanup" Nov 01 06:47:05.895031 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895015 4385 cpu_manager.go:414] "containerMap: removing container" podUID="8cce3484-b9d3-4d52-a06d-1f407c3a96c4" containerName="cleanup" Nov 01 06:47:05.895031 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895020 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cce3484-b9d3-4d52-a06d-1f407c3a96c4" containerName="cleanup" Nov 01 06:47:05.895031 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895029 4385 cpu_manager.go:414] "containerMap: removing container" podUID="5450017c-fdb1-448a-a82d-593f799789c4" containerName="cleanup" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895034 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5450017c-fdb1-448a-a82d-593f799789c4" containerName="cleanup" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895041 4385 cpu_manager.go:414] "containerMap: removing container" podUID="8cce3484-b9d3-4d52-a06d-1f407c3a96c4" containerName="cleanup" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895045 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cce3484-b9d3-4d52-a06d-1f407c3a96c4" containerName="cleanup" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895052 4385 cpu_manager.go:414] "containerMap: removing container" podUID="5450017c-fdb1-448a-a82d-593f799789c4" containerName="cleanup" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895056 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5450017c-fdb1-448a-a82d-593f799789c4" containerName="cleanup" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895069 4385 cpu_manager.go:414] "containerMap: removing container" podUID="73f8815d-000f-4134-9169-6b686f776e58" containerName="cleanup" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895073 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f8815d-000f-4134-9169-6b686f776e58" containerName="cleanup" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895080 4385 cpu_manager.go:414] "containerMap: removing container" podUID="73f8815d-000f-4134-9169-6b686f776e58" containerName="cleanup" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895084 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f8815d-000f-4134-9169-6b686f776e58" containerName="cleanup" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895095 4385 cpu_manager.go:414] "containerMap: removing container" podUID="5450017c-fdb1-448a-a82d-593f799789c4" containerName="cleanup" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895099 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5450017c-fdb1-448a-a82d-593f799789c4" containerName="cleanup" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895108 4385 cpu_manager.go:414] "containerMap: removing container" podUID="e107c800-1601-48ae-b4ea-8606ec6c1b67" containerName="maas-api" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895113 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e107c800-1601-48ae-b4ea-8606ec6c1b67" containerName="maas-api" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895183 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="5450017c-fdb1-448a-a82d-593f799789c4" containerName="cleanup" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895190 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cce3484-b9d3-4d52-a06d-1f407c3a96c4" containerName="cleanup" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895197 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="5450017c-fdb1-448a-a82d-593f799789c4" containerName="cleanup" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895204 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f8815d-000f-4134-9169-6b686f776e58" containerName="cleanup" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895210 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f8815d-000f-4134-9169-6b686f776e58" containerName="cleanup" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895215 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="924d1205-076c-4787-a9ca-8291af9115a4" containerName="cleanup" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895222 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="5450017c-fdb1-448a-a82d-593f799789c4" containerName="cleanup" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895228 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="e107c800-1601-48ae-b4ea-8606ec6c1b67" containerName="maas-api" Nov 01 06:47:05.895226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895234 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f8815d-000f-4134-9169-6b686f776e58" containerName="cleanup" Nov 01 06:47:05.895891 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895240 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="924d1205-076c-4787-a9ca-8291af9115a4" containerName="cleanup" Nov 01 06:47:05.895891 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895247 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="924d1205-076c-4787-a9ca-8291af9115a4" containerName="cleanup" Nov 01 06:47:05.895891 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.895253 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cce3484-b9d3-4d52-a06d-1f407c3a96c4" containerName="cleanup" Nov 01 06:47:05.899557 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.899542 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" Nov 01 06:47:05.919738 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.919710 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrllt\" (UniqueName: \"kubernetes.io/projected/705ab665-d772-4ba3-af1a-56885b1f1ff7-kube-api-access-lrllt\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh\" (UID: \"705ab665-d772-4ba3-af1a-56885b1f1ff7\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" Nov 01 06:47:05.919896 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.919761 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/705ab665-d772-4ba3-af1a-56885b1f1ff7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh\" (UID: \"705ab665-d772-4ba3-af1a-56885b1f1ff7\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" Nov 01 06:47:05.919896 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.919878 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705ab665-d772-4ba3-af1a-56885b1f1ff7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh\" (UID: \"705ab665-d772-4ba3-af1a-56885b1f1ff7\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" Nov 01 06:47:05.919999 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:05.919907 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/705ab665-d772-4ba3-af1a-56885b1f1ff7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh\" (UID: \"705ab665-d772-4ba3-af1a-56885b1f1ff7\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" Nov 01 06:47:06.002024 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.001949 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-120370fe39d3\"" Nov 01 06:47:06.002024 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.001949 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-a83cd424b5a5\"" Nov 01 06:47:06.002024 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.001949 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"routing-overlay\"" Nov 01 06:47:06.002024 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.001957 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"openshift-service-ca.crt\"" Nov 01 06:47:06.002024 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.001960 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-2b38f786e9be\"" Nov 01 06:47:06.002444 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.002025 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"payload-processing-external-model-plugins-e2e-extmod-b-dc6e48\"" Nov 01 06:47:06.002444 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.002057 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"payload-processing-external-model-e2e--958558de-dockercfg-qjzd6\"" Nov 01 06:47:06.002444 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.002116 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"kube-root-ca.crt\"" Nov 01 06:47:06.005555 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.005532 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh"] Nov 01 06:47:06.021116 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.021091 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrllt\" (UniqueName: \"kubernetes.io/projected/705ab665-d772-4ba3-af1a-56885b1f1ff7-kube-api-access-lrllt\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh\" (UID: \"705ab665-d772-4ba3-af1a-56885b1f1ff7\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" Nov 01 06:47:06.021239 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.021136 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/705ab665-d772-4ba3-af1a-56885b1f1ff7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh\" (UID: \"705ab665-d772-4ba3-af1a-56885b1f1ff7\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" Nov 01 06:47:06.021334 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.021316 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705ab665-d772-4ba3-af1a-56885b1f1ff7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh\" (UID: \"705ab665-d772-4ba3-af1a-56885b1f1ff7\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" Nov 01 06:47:06.021389 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.021365 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/705ab665-d772-4ba3-af1a-56885b1f1ff7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh\" (UID: \"705ab665-d772-4ba3-af1a-56885b1f1ff7\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" Nov 01 06:47:06.022071 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.022050 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705ab665-d772-4ba3-af1a-56885b1f1ff7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh\" (UID: \"705ab665-d772-4ba3-af1a-56885b1f1ff7\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" Nov 01 06:47:06.022159 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.022105 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/705ab665-d772-4ba3-af1a-56885b1f1ff7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh\" (UID: \"705ab665-d772-4ba3-af1a-56885b1f1ff7\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" Nov 01 06:47:06.024210 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.024187 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/705ab665-d772-4ba3-af1a-56885b1f1ff7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh\" (UID: \"705ab665-d772-4ba3-af1a-56885b1f1ff7\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" Nov 01 06:47:06.077794 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.077757 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrllt\" (UniqueName: \"kubernetes.io/projected/705ab665-d772-4ba3-af1a-56885b1f1ff7-kube-api-access-lrllt\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh\" (UID: \"705ab665-d772-4ba3-af1a-56885b1f1ff7\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" Nov 01 06:47:06.219020 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.217246 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" Nov 01 06:47:06.599726 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.599703 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh"] Nov 01 06:47:06.601105 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:47:06.601075 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705ab665_d772_4ba3_af1a_56885b1f1ff7.slice/crio-4d16eb761a1c88e99eed7335d6e8fbbadaff3699ac09ac65bf5f3656ca50838c WatchSource:0}: Error finding container 4d16eb761a1c88e99eed7335d6e8fbbadaff3699ac09ac65bf5f3656ca50838c: Status 404 returned error can't find the container with id 4d16eb761a1c88e99eed7335d6e8fbbadaff3699ac09ac65bf5f3656ca50838c Nov 01 06:47:06.780533 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:47:06.780498 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:47:06.925817 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.925724 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" event={"ID":"705ab665-d772-4ba3-af1a-56885b1f1ff7","Type":"ContainerStarted","Data":"a2ce75c1f580ebcf29e05979bb3a566745c02ca156cbda2c40f316d2bbbe1b68"} Nov 01 06:47:06.925817 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.925788 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" event={"ID":"705ab665-d772-4ba3-af1a-56885b1f1ff7","Type":"ContainerStarted","Data":"4d16eb761a1c88e99eed7335d6e8fbbadaff3699ac09ac65bf5f3656ca50838c"} Nov 01 06:47:06.926248 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:06.925936 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" Nov 01 06:47:07.042755 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:07.042707 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" podStartSLOduration=2.042688927 podStartE2EDuration="2.042688927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:47:07.038522558 +0000 UTC m=+2236.802322415" watchObservedRunningTime="2026-11-01 06:47:07.042688927 +0000 UTC m=+2236.806488811" Nov 01 06:47:08.371743 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:08.371707 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:10.425177 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:10.425139 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:10.640141 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:10.640098 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:11.143606 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:11.143563 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:11.321021 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:11.320984 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:11.692612 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:11.692576 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j"] Nov 01 06:47:11.693454 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:11.693433 4385 cpu_manager.go:414] "containerMap: removing container" podUID="924d1205-076c-4787-a9ca-8291af9115a4" containerName="cleanup" Nov 01 06:47:11.693534 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:11.693458 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924d1205-076c-4787-a9ca-8291af9115a4" containerName="cleanup" Nov 01 06:47:11.693534 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:11.693493 4385 cpu_manager.go:414] "containerMap: removing container" podUID="8cce3484-b9d3-4d52-a06d-1f407c3a96c4" containerName="cleanup" Nov 01 06:47:11.693534 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:11.693502 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cce3484-b9d3-4d52-a06d-1f407c3a96c4" containerName="cleanup" Nov 01 06:47:11.693534 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:11.693526 4385 cpu_manager.go:414] "containerMap: removing container" podUID="73f8815d-000f-4134-9169-6b686f776e58" containerName="cleanup" Nov 01 06:47:11.693659 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:11.693535 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f8815d-000f-4134-9169-6b686f776e58" containerName="cleanup" Nov 01 06:47:11.693695 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:11.693669 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cce3484-b9d3-4d52-a06d-1f407c3a96c4" containerName="cleanup" Nov 01 06:47:11.698816 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:11.698793 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j" Nov 01 06:47:11.751466 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:11.751441 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j"] Nov 01 06:47:11.793334 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:11.793298 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t68jg\" (UniqueName: \"kubernetes.io/projected/91f7307b-e9c5-45ee-871c-d1de04507c0a-kube-api-access-t68jg\") pod \"payload-processing-e2e-api-b-d855db-76f476689-zjv2j\" (UID: \"91f7307b-e9c5-45ee-871c-d1de04507c0a\") " pod="openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j" Nov 01 06:47:11.793495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:11.793369 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91f7307b-e9c5-45ee-871c-d1de04507c0a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d855db-76f476689-zjv2j\" (UID: \"91f7307b-e9c5-45ee-871c-d1de04507c0a\") " pod="openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j" Nov 01 06:47:11.796139 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:11.796095 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d855db\"" Nov 01 06:47:11.796352 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:11.796329 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d855db-dockercfg-sfvmq\"" Nov 01 06:47:11.895140 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:11.895100 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t68jg\" (UniqueName: \"kubernetes.io/projected/91f7307b-e9c5-45ee-871c-d1de04507c0a-kube-api-access-t68jg\") pod \"payload-processing-e2e-api-b-d855db-76f476689-zjv2j\" (UID: \"91f7307b-e9c5-45ee-871c-d1de04507c0a\") " pod="openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j" Nov 01 06:47:11.895347 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:11.895170 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91f7307b-e9c5-45ee-871c-d1de04507c0a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d855db-76f476689-zjv2j\" (UID: \"91f7307b-e9c5-45ee-871c-d1de04507c0a\") " pod="openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j" Nov 01 06:47:11.896114 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:11.896089 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91f7307b-e9c5-45ee-871c-d1de04507c0a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d855db-76f476689-zjv2j\" (UID: \"91f7307b-e9c5-45ee-871c-d1de04507c0a\") " pod="openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j" Nov 01 06:47:11.952070 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:11.952003 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t68jg\" (UniqueName: \"kubernetes.io/projected/91f7307b-e9c5-45ee-871c-d1de04507c0a-kube-api-access-t68jg\") pod \"payload-processing-e2e-api-b-d855db-76f476689-zjv2j\" (UID: \"91f7307b-e9c5-45ee-871c-d1de04507c0a\") " pod="openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j" Nov 01 06:47:12.012982 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:12.012959 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j" Nov 01 06:47:12.389466 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:12.389441 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j"] Nov 01 06:47:12.390337 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:47:12.390312 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f7307b_e9c5_45ee_871c_d1de04507c0a.slice/crio-250ab6cb1c1353469922a1d25ccd38ec33a01f09665b9c22afd73fcfd4aac4a9 WatchSource:0}: Error finding container 250ab6cb1c1353469922a1d25ccd38ec33a01f09665b9c22afd73fcfd4aac4a9: Status 404 returned error can't find the container with id 250ab6cb1c1353469922a1d25ccd38ec33a01f09665b9c22afd73fcfd4aac4a9 Nov 01 06:47:12.981592 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:12.981553 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j" event={"ID":"91f7307b-e9c5-45ee-871c-d1de04507c0a","Type":"ContainerStarted","Data":"d614f9ac349a5f1c1ead58a1f9ab330f50da70759400c9138f5ef3ce6aaaa40f"} Nov 01 06:47:12.982016 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:12.981606 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j" event={"ID":"91f7307b-e9c5-45ee-871c-d1de04507c0a","Type":"ContainerStarted","Data":"250ab6cb1c1353469922a1d25ccd38ec33a01f09665b9c22afd73fcfd4aac4a9"} Nov 01 06:47:12.982016 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:12.981713 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j" Nov 01 06:47:13.096298 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:13.096247 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j" podStartSLOduration=2.096232369 podStartE2EDuration="2.096232369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:47:13.092518868 +0000 UTC m=+2242.856318724" watchObservedRunningTime="2026-11-01 06:47:13.096232369 +0000 UTC m=+2242.860032235" Nov 01 06:47:13.324609 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:13.324557 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:17.943299 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:17.943246 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" podUID="705ab665-d772-4ba3-af1a-56885b1f1ff7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:47:18.115534 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:18.115501 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:22.473358 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:22.473324 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9"] Nov 01 06:47:22.477609 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:22.477586 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" Nov 01 06:47:22.530350 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:22.530325 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-caf74d-serving-cert\"" Nov 01 06:47:22.585427 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:22.585394 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9"] Nov 01 06:47:22.610114 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:22.610081 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-metrics-certs\") pod \"maas-api-e2e-extmod-a-caf74d-744456f584-8vct9\" (UID: \"0b9a9dca-7256-4ff9-9bab-dc46d8198aba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" Nov 01 06:47:22.610406 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:22.610380 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc25z\" (UniqueName: \"kubernetes.io/projected/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-kube-api-access-jc25z\") pod \"maas-api-e2e-extmod-a-caf74d-744456f584-8vct9\" (UID: \"0b9a9dca-7256-4ff9-9bab-dc46d8198aba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" Nov 01 06:47:22.610555 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:22.610423 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-caf74d-744456f584-8vct9\" (UID: \"0b9a9dca-7256-4ff9-9bab-dc46d8198aba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" Nov 01 06:47:22.610555 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:22.610503 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-maas-api-tls\") pod \"maas-api-e2e-extmod-a-caf74d-744456f584-8vct9\" (UID: \"0b9a9dca-7256-4ff9-9bab-dc46d8198aba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" Nov 01 06:47:22.684559 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:22.684526 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:22.711725 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:22.711695 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-maas-api-tls\") pod \"maas-api-e2e-extmod-a-caf74d-744456f584-8vct9\" (UID: \"0b9a9dca-7256-4ff9-9bab-dc46d8198aba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" Nov 01 06:47:22.711867 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:22.711822 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-metrics-certs\") pod \"maas-api-e2e-extmod-a-caf74d-744456f584-8vct9\" (UID: \"0b9a9dca-7256-4ff9-9bab-dc46d8198aba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" Nov 01 06:47:22.711914 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:47:22.711893 4385 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-serving-cert: secret "maas-api-e2e-extmod-a-caf74d-serving-cert" not found Nov 01 06:47:22.711914 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:22.711896 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc25z\" (UniqueName: \"kubernetes.io/projected/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-kube-api-access-jc25z\") pod \"maas-api-e2e-extmod-a-caf74d-744456f584-8vct9\" (UID: \"0b9a9dca-7256-4ff9-9bab-dc46d8198aba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" Nov 01 06:47:22.711986 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:22.711928 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-caf74d-744456f584-8vct9\" (UID: \"0b9a9dca-7256-4ff9-9bab-dc46d8198aba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" Nov 01 06:47:22.711986 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:47:22.711955 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-maas-api-tls podName:0b9a9dca-7256-4ff9-9bab-dc46d8198aba nodeName:}" failed. No retries permitted until 2026-11-01 06:47:23.211933983 +0000 UTC m=+2252.975733822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-maas-api-tls") pod "maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" (UID: "0b9a9dca-7256-4ff9-9bab-dc46d8198aba") : secret "maas-api-e2e-extmod-a-caf74d-serving-cert" not found Nov 01 06:47:22.712510 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:22.712488 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-caf74d-744456f584-8vct9\" (UID: \"0b9a9dca-7256-4ff9-9bab-dc46d8198aba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" Nov 01 06:47:22.714642 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:22.714618 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-metrics-certs\") pod \"maas-api-e2e-extmod-a-caf74d-744456f584-8vct9\" (UID: \"0b9a9dca-7256-4ff9-9bab-dc46d8198aba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" Nov 01 06:47:22.769087 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:22.769012 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc25z\" (UniqueName: \"kubernetes.io/projected/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-kube-api-access-jc25z\") pod \"maas-api-e2e-extmod-a-caf74d-744456f584-8vct9\" (UID: \"0b9a9dca-7256-4ff9-9bab-dc46d8198aba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" Nov 01 06:47:22.879404 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:22.879317 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:23.021526 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:23.021434 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:23.217364 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:23.217329 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-maas-api-tls\") pod \"maas-api-e2e-extmod-a-caf74d-744456f584-8vct9\" (UID: \"0b9a9dca-7256-4ff9-9bab-dc46d8198aba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" Nov 01 06:47:23.220107 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:23.220081 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-maas-api-tls\") pod \"maas-api-e2e-extmod-a-caf74d-744456f584-8vct9\" (UID: \"0b9a9dca-7256-4ff9-9bab-dc46d8198aba\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" Nov 01 06:47:23.392444 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:23.392415 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" Nov 01 06:47:23.775690 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:47:23.775650 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9a9dca_7256_4ff9_9bab_dc46d8198aba.slice/crio-1f2ae14d7fc1dc05bc59b20f94a3336445b8c70073c082f8823c038b27c82f9a WatchSource:0}: Error finding container 1f2ae14d7fc1dc05bc59b20f94a3336445b8c70073c082f8823c038b27c82f9a: Status 404 returned error can't find the container with id 1f2ae14d7fc1dc05bc59b20f94a3336445b8c70073c082f8823c038b27c82f9a Nov 01 06:47:23.777169 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:23.777143 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9"] Nov 01 06:47:23.777273 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:23.777196 4385 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:47:23.992479 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:23.992451 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j" Nov 01 06:47:24.055315 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:24.055231 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" event={"ID":"0b9a9dca-7256-4ff9-9bab-dc46d8198aba","Type":"ContainerStarted","Data":"1f2ae14d7fc1dc05bc59b20f94a3336445b8c70073c082f8823c038b27c82f9a"} Nov 01 06:47:25.067560 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:25.067473 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" event={"ID":"0b9a9dca-7256-4ff9-9bab-dc46d8198aba","Type":"ContainerStarted","Data":"8ec5d170b1449f41b0b76dfdfc3bf109c18e9e3846d4a8e5d430af27eec68ad2"} Nov 01 06:47:25.067560 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:25.067534 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" Nov 01 06:47:25.185086 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:25.185035 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" podStartSLOduration=2.259054007 podStartE2EDuration="3.185020676s" totalImagesPullingTime="925.966669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:47:25.182080653 +0000 UTC m=+2254.945880510" watchObservedRunningTime="2026-11-01 06:47:25.185020676 +0000 UTC m=+2254.948820570" Nov 01 06:47:27.530399 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:27.530359 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:27.941431 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:27.941400 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" podUID="705ab665-d772-4ba3-af1a-56885b1f1ff7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:47:31.079768 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:31.079738 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" Nov 01 06:47:31.201771 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:31.201675 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:31.345284 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:31.345196 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:32.059300 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:32.059267 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:32.325521 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:32.325490 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:35.261154 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:35.261115 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:36.785078 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:47:36.785038 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:47:36.787367 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:36.787337 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:36.929849 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:36.929803 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:37.080227 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:37.080195 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:37.940793 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:37.940758 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" podUID="705ab665-d772-4ba3-af1a-56885b1f1ff7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:47:39.638892 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:39.638861 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl"] Nov 01 06:47:39.643064 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:39.643045 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" Nov 01 06:47:39.710440 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:39.710410 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec39162f-e923-4d1f-b491-529ba0b08a1c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl\" (UID: \"ec39162f-e923-4d1f-b491-529ba0b08a1c\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" Nov 01 06:47:39.710605 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:39.710581 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec39162f-e923-4d1f-b491-529ba0b08a1c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl\" (UID: \"ec39162f-e923-4d1f-b491-529ba0b08a1c\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" Nov 01 06:47:39.710666 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:39.710631 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blwp7\" (UniqueName: \"kubernetes.io/projected/ec39162f-e923-4d1f-b491-529ba0b08a1c-kube-api-access-blwp7\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl\" (UID: \"ec39162f-e923-4d1f-b491-529ba0b08a1c\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" Nov 01 06:47:39.710666 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:39.710647 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec39162f-e923-4d1f-b491-529ba0b08a1c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl\" (UID: \"ec39162f-e923-4d1f-b491-529ba0b08a1c\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" Nov 01 06:47:39.748155 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:39.748123 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl"] Nov 01 06:47:39.811604 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:39.811561 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blwp7\" (UniqueName: \"kubernetes.io/projected/ec39162f-e923-4d1f-b491-529ba0b08a1c-kube-api-access-blwp7\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl\" (UID: \"ec39162f-e923-4d1f-b491-529ba0b08a1c\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" Nov 01 06:47:39.811604 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:39.811602 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec39162f-e923-4d1f-b491-529ba0b08a1c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl\" (UID: \"ec39162f-e923-4d1f-b491-529ba0b08a1c\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" Nov 01 06:47:39.811871 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:39.811643 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec39162f-e923-4d1f-b491-529ba0b08a1c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl\" (UID: \"ec39162f-e923-4d1f-b491-529ba0b08a1c\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" Nov 01 06:47:39.811871 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:39.811848 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec39162f-e923-4d1f-b491-529ba0b08a1c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl\" (UID: \"ec39162f-e923-4d1f-b491-529ba0b08a1c\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" Nov 01 06:47:39.812523 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:39.812453 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec39162f-e923-4d1f-b491-529ba0b08a1c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl\" (UID: \"ec39162f-e923-4d1f-b491-529ba0b08a1c\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" Nov 01 06:47:39.812705 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:39.812645 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec39162f-e923-4d1f-b491-529ba0b08a1c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl\" (UID: \"ec39162f-e923-4d1f-b491-529ba0b08a1c\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" Nov 01 06:47:39.814722 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:39.814697 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec39162f-e923-4d1f-b491-529ba0b08a1c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl\" (UID: \"ec39162f-e923-4d1f-b491-529ba0b08a1c\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" Nov 01 06:47:39.867631 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:39.867598 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blwp7\" (UniqueName: \"kubernetes.io/projected/ec39162f-e923-4d1f-b491-529ba0b08a1c-kube-api-access-blwp7\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl\" (UID: \"ec39162f-e923-4d1f-b491-529ba0b08a1c\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" Nov 01 06:47:39.957134 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:39.957062 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" Nov 01 06:47:40.335604 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:40.335574 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl"] Nov 01 06:47:40.335745 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:47:40.335718 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec39162f_e923_4d1f_b491_529ba0b08a1c.slice/crio-89f3b25b0cc6ef9b061e6d159a055d5e24ddece3b2efd4ce24d50923606402a2 WatchSource:0}: Error finding container 89f3b25b0cc6ef9b061e6d159a055d5e24ddece3b2efd4ce24d50923606402a2: Status 404 returned error can't find the container with id 89f3b25b0cc6ef9b061e6d159a055d5e24ddece3b2efd4ce24d50923606402a2 Nov 01 06:47:41.181109 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:41.181065 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" event={"ID":"ec39162f-e923-4d1f-b491-529ba0b08a1c","Type":"ContainerStarted","Data":"70546a6a52d6094987b06e9069b8fb9341db1d28f79c1fbcd286f381bee2e985"} Nov 01 06:47:41.181606 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:41.181138 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" event={"ID":"ec39162f-e923-4d1f-b491-529ba0b08a1c","Type":"ContainerStarted","Data":"89f3b25b0cc6ef9b061e6d159a055d5e24ddece3b2efd4ce24d50923606402a2"} Nov 01 06:47:41.181606 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:41.181173 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" Nov 01 06:47:41.295980 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:41.295922 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" podStartSLOduration=2.295907971 podStartE2EDuration="2.295907971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:47:41.292819609 +0000 UTC m=+2271.056619476" watchObservedRunningTime="2026-11-01 06:47:41.295907971 +0000 UTC m=+2271.059707873" Nov 01 06:47:41.934018 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:41.933974 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:44.806713 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:44.806631 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:45.007643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:45.007606 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:46.132989 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:46.132947 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:46.650703 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:46.650669 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:46.952559 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:46.952466 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:47.104193 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:47.104140 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:47.601109 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:47.601070 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:47.941216 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:47.941125 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" podUID="705ab665-d772-4ba3-af1a-56885b1f1ff7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:47:48.229897 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:48.229795 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:48.378139 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:48.377585 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:48.634850 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:48.634801 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:49.490622 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:49.490584 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:51.862918 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:51.862882 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:52.190815 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:52.190728 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" podUID="ec39162f-e923-4d1f-b491-529ba0b08a1c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:47:53.739463 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:53.739428 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:56.523209 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:56.523175 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:47:57.940777 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:47:57.940735 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" podUID="705ab665-d772-4ba3-af1a-56885b1f1ff7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:47:59.560417 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:47:59.560383 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-25719d/agc-cred-c7ba2136e663: secret "agc-cred-c7ba2136e663" not found Nov 01 06:47:59.560417 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:47:59.560414 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-25719d/agc-cred-f6f2772a70c5: secret "agc-cred-f6f2772a70c5" not found Nov 01 06:47:59.560417 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:47:59.560428 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk: [secret "agc-cred-c7ba2136e663" not found, secret "agc-cred-f6f2772a70c5" not found] Nov 01 06:47:59.561072 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:47:59.560511 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-provider-credentials podName:46bcb2e8-bdb2-4632-911d-d0668846115a nodeName:}" failed. No retries permitted until 2026-11-01 06:48:00.060488426 +0000 UTC m=+2289.824288276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" (UID: "46bcb2e8-bdb2-4632-911d-d0668846115a") : [secret "agc-cred-c7ba2136e663" not found, secret "agc-cred-f6f2772a70c5" not found] Nov 01 06:48:00.066759 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:00.066726 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-25719d/agc-cred-c7ba2136e663: secret "agc-cred-c7ba2136e663" not found Nov 01 06:48:00.066759 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:00.066757 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-25719d/agc-cred-f6f2772a70c5: secret "agc-cred-f6f2772a70c5" not found Nov 01 06:48:00.066759 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:00.066766 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk: [secret "agc-cred-c7ba2136e663" not found, secret "agc-cred-f6f2772a70c5" not found] Nov 01 06:48:00.067126 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:00.066857 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-provider-credentials podName:46bcb2e8-bdb2-4632-911d-d0668846115a nodeName:}" failed. No retries permitted until 2026-11-01 06:48:01.066816222 +0000 UTC m=+2290.830616069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" (UID: "46bcb2e8-bdb2-4632-911d-d0668846115a") : [secret "agc-cred-c7ba2136e663" not found, secret "agc-cred-f6f2772a70c5" not found] Nov 01 06:48:00.573428 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:00.573394 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-841de3b011bd: secret "agc-cred-841de3b011bd" not found Nov 01 06:48:00.574011 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:00.573987 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-c3e56f59453b: secret "agc-cred-c3e56f59453b" not found Nov 01 06:48:00.574211 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:00.574192 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc: [secret "agc-cred-841de3b011bd" not found, secret "agc-cred-c3e56f59453b" not found] Nov 01 06:48:00.574373 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:00.574356 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-provider-credentials podName:e5a2a7f9-63f8-44ae-a226-c5fd9574643f nodeName:}" failed. No retries permitted until 2026-11-01 06:48:01.07433766 +0000 UTC m=+2290.838137508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" (UID: "e5a2a7f9-63f8-44ae-a226-c5fd9574643f") : [secret "agc-cred-841de3b011bd" not found, secret "agc-cred-c3e56f59453b" not found] Nov 01 06:48:01.080884 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:01.080824 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-841de3b011bd: secret "agc-cred-841de3b011bd" not found Nov 01 06:48:01.080884 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:01.080870 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-c3e56f59453b: secret "agc-cred-c3e56f59453b" not found Nov 01 06:48:01.080884 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:01.080879 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc: [secret "agc-cred-841de3b011bd" not found, secret "agc-cred-c3e56f59453b" not found] Nov 01 06:48:01.081205 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:01.080890 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-25719d/agc-cred-c7ba2136e663: secret "agc-cred-c7ba2136e663" not found Nov 01 06:48:01.081205 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:01.080913 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-25719d/agc-cred-f6f2772a70c5: secret "agc-cred-f6f2772a70c5" not found Nov 01 06:48:01.081205 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:01.080925 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk: [secret "agc-cred-c7ba2136e663" not found, secret "agc-cred-f6f2772a70c5" not found] Nov 01 06:48:01.081205 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:01.080937 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-provider-credentials podName:e5a2a7f9-63f8-44ae-a226-c5fd9574643f nodeName:}" failed. No retries permitted until 2026-11-01 06:48:02.080917223 +0000 UTC m=+2291.844717058 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" (UID: "e5a2a7f9-63f8-44ae-a226-c5fd9574643f") : [secret "agc-cred-841de3b011bd" not found, secret "agc-cred-c3e56f59453b" not found] Nov 01 06:48:01.081205 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:01.080974 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-provider-credentials podName:46bcb2e8-bdb2-4632-911d-d0668846115a nodeName:}" failed. No retries permitted until 2026-11-01 06:48:03.080960865 +0000 UTC m=+2292.844760701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" (UID: "46bcb2e8-bdb2-4632-911d-d0668846115a") : [secret "agc-cred-c7ba2136e663" not found, secret "agc-cred-f6f2772a70c5" not found] Nov 01 06:48:01.219546 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:01.219512 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:01.932985 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:01.932945 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:02.095917 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:02.095885 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-841de3b011bd: secret "agc-cred-841de3b011bd" not found Nov 01 06:48:02.096186 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:02.096169 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-97c3302b7889: secret "agc-cred-97c3302b7889" not found Nov 01 06:48:02.096327 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:02.096318 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-c3e56f59453b: secret "agc-cred-c3e56f59453b" not found Nov 01 06:48:02.096429 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:02.096419 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc: [secret "agc-cred-841de3b011bd" not found, secret "agc-cred-97c3302b7889" not found, secret "agc-cred-c3e56f59453b" not found] Nov 01 06:48:02.096596 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:02.096582 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-provider-credentials podName:e5a2a7f9-63f8-44ae-a226-c5fd9574643f nodeName:}" failed. No retries permitted until 2026-11-01 06:48:04.096559521 +0000 UTC m=+2293.860359375 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" (UID: "e5a2a7f9-63f8-44ae-a226-c5fd9574643f") : [secret "agc-cred-841de3b011bd" not found, secret "agc-cred-97c3302b7889" not found, secret "agc-cred-c3e56f59453b" not found] Nov 01 06:48:02.135599 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:02.135567 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:02.189954 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:02.189822 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" podUID="ec39162f-e923-4d1f-b491-529ba0b08a1c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:48:02.553184 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:02.553076 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:02.699437 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:02.699396 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:02.984214 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:02.984172 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:03.109343 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:03.109308 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-25719d/agc-cred-c7ba2136e663: secret "agc-cred-c7ba2136e663" not found Nov 01 06:48:03.109343 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:03.109341 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-25719d/agc-cred-d41a0e39961b: secret "agc-cred-d41a0e39961b" not found Nov 01 06:48:03.109343 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:03.109348 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-25719d/agc-cred-f6f2772a70c5: secret "agc-cred-f6f2772a70c5" not found Nov 01 06:48:03.109726 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:03.109360 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk: [secret "agc-cred-c7ba2136e663" not found, secret "agc-cred-d41a0e39961b" not found, secret "agc-cred-f6f2772a70c5" not found] Nov 01 06:48:03.109726 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:03.109425 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-provider-credentials podName:46bcb2e8-bdb2-4632-911d-d0668846115a nodeName:}" failed. No retries permitted until 2026-11-01 06:48:07.10940558 +0000 UTC m=+2296.873205417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" (UID: "46bcb2e8-bdb2-4632-911d-d0668846115a") : [secret "agc-cred-c7ba2136e663" not found, secret "agc-cred-d41a0e39961b" not found, secret "agc-cred-f6f2772a70c5" not found] Nov 01 06:48:04.120795 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:04.120751 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-841de3b011bd: secret "agc-cred-841de3b011bd" not found Nov 01 06:48:04.120795 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:04.120784 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-97c3302b7889: secret "agc-cred-97c3302b7889" not found Nov 01 06:48:04.120795 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:04.120793 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-c3e56f59453b: secret "agc-cred-c3e56f59453b" not found Nov 01 06:48:04.120795 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:04.120803 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc: [secret "agc-cred-841de3b011bd" not found, secret "agc-cred-97c3302b7889" not found, secret "agc-cred-c3e56f59453b" not found] Nov 01 06:48:04.121438 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:04.120914 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-provider-credentials podName:e5a2a7f9-63f8-44ae-a226-c5fd9574643f nodeName:}" failed. No retries permitted until 2026-11-01 06:48:08.120892764 +0000 UTC m=+2297.884692599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" (UID: "e5a2a7f9-63f8-44ae-a226-c5fd9574643f") : [secret "agc-cred-841de3b011bd" not found, secret "agc-cred-97c3302b7889" not found, secret "agc-cred-c3e56f59453b" not found] Nov 01 06:48:05.324274 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:05.324240 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl"] Nov 01 06:48:05.325566 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:05.325541 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" podUID="ec39162f-e923-4d1f-b491-529ba0b08a1c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:48:05.684709 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:05.684626 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp"] Nov 01 06:48:05.689773 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:05.689756 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" Nov 01 06:48:05.740477 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:05.740442 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-maas-api-tls\") pod \"maas-api-e2e-api-b-d855db-774554c59-jkjfp\" (UID: \"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" Nov 01 06:48:05.740683 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:05.740494 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gvld\" (UniqueName: \"kubernetes.io/projected/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-kube-api-access-8gvld\") pod \"maas-api-e2e-api-b-d855db-774554c59-jkjfp\" (UID: \"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" Nov 01 06:48:05.740757 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:05.740728 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-metrics-certs\") pod \"maas-api-e2e-api-b-d855db-774554c59-jkjfp\" (UID: \"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" Nov 01 06:48:05.740826 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:05.740813 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-openshift-service-ca\") pod \"maas-api-e2e-api-b-d855db-774554c59-jkjfp\" (UID: \"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" Nov 01 06:48:05.743406 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:05.743365 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d855db-serving-cert\"" Nov 01 06:48:05.746183 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:05.746163 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp"] Nov 01 06:48:05.842676 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:05.842635 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-maas-api-tls\") pod \"maas-api-e2e-api-b-d855db-774554c59-jkjfp\" (UID: \"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" Nov 01 06:48:05.843106 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:05.842724 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gvld\" (UniqueName: \"kubernetes.io/projected/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-kube-api-access-8gvld\") pod \"maas-api-e2e-api-b-d855db-774554c59-jkjfp\" (UID: \"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" Nov 01 06:48:05.843106 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:05.842800 4385 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-serving-cert: secret "maas-api-e2e-api-b-d855db-serving-cert" not found Nov 01 06:48:05.843106 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:05.842914 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-maas-api-tls podName:eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7 nodeName:}" failed. No retries permitted until 2026-11-01 06:48:06.342888849 +0000 UTC m=+2296.106688695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-maas-api-tls") pod "maas-api-e2e-api-b-d855db-774554c59-jkjfp" (UID: "eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7") : secret "maas-api-e2e-api-b-d855db-serving-cert" not found Nov 01 06:48:05.843106 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:05.842956 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-metrics-certs\") pod \"maas-api-e2e-api-b-d855db-774554c59-jkjfp\" (UID: \"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" Nov 01 06:48:05.843106 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:05.843025 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-openshift-service-ca\") pod \"maas-api-e2e-api-b-d855db-774554c59-jkjfp\" (UID: \"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" Nov 01 06:48:05.843643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:05.843621 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-openshift-service-ca\") pod \"maas-api-e2e-api-b-d855db-774554c59-jkjfp\" (UID: \"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" Nov 01 06:48:05.845932 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:05.845911 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-metrics-certs\") pod \"maas-api-e2e-api-b-d855db-774554c59-jkjfp\" (UID: \"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" Nov 01 06:48:05.883036 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:05.883001 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:05.901955 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:05.901925 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gvld\" (UniqueName: \"kubernetes.io/projected/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-kube-api-access-8gvld\") pod \"maas-api-e2e-api-b-d855db-774554c59-jkjfp\" (UID: \"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" Nov 01 06:48:06.077142 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:06.077110 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:06.361887 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:06.361756 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-maas-api-tls\") pod \"maas-api-e2e-api-b-d855db-774554c59-jkjfp\" (UID: \"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" Nov 01 06:48:06.369610 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:06.369582 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-maas-api-tls\") pod \"maas-api-e2e-api-b-d855db-774554c59-jkjfp\" (UID: \"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" Nov 01 06:48:06.603983 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:06.603947 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" Nov 01 06:48:06.781111 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:06.781072 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:48:06.983119 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:48:06.983089 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaaaefbd_b8c6_4179_8ce1_f492ccfde1f7.slice/crio-1392fdf1903d0668db14c5cf457266044eb484b1d71e75cf643433dd25d09ef9 WatchSource:0}: Error finding container 1392fdf1903d0668db14c5cf457266044eb484b1d71e75cf643433dd25d09ef9: Status 404 returned error can't find the container with id 1392fdf1903d0668db14c5cf457266044eb484b1d71e75cf643433dd25d09ef9 Nov 01 06:48:06.983281 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:06.983262 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp"] Nov 01 06:48:07.171270 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:07.171235 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-25719d/agc-cred-c7ba2136e663: secret "agc-cred-c7ba2136e663" not found Nov 01 06:48:07.171270 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:07.171264 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-25719d/agc-cred-d41a0e39961b: secret "agc-cred-d41a0e39961b" not found Nov 01 06:48:07.171270 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:07.171276 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-25719d/agc-cred-f6f2772a70c5: secret "agc-cred-f6f2772a70c5" not found Nov 01 06:48:07.171574 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:07.171288 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk: [secret "agc-cred-c7ba2136e663" not found, secret "agc-cred-d41a0e39961b" not found, secret "agc-cred-f6f2772a70c5" not found] Nov 01 06:48:07.171574 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:07.171369 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-provider-credentials podName:46bcb2e8-bdb2-4632-911d-d0668846115a nodeName:}" failed. No retries permitted until 2026-11-01 06:48:15.171349841 +0000 UTC m=+2304.935149678 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" (UID: "46bcb2e8-bdb2-4632-911d-d0668846115a") : [secret "agc-cred-c7ba2136e663" not found, secret "agc-cred-d41a0e39961b" not found, secret "agc-cred-f6f2772a70c5" not found] Nov 01 06:48:07.382922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:07.382881 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" event={"ID":"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7","Type":"ContainerStarted","Data":"1392fdf1903d0668db14c5cf457266044eb484b1d71e75cf643433dd25d09ef9"} Nov 01 06:48:07.941293 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:07.941257 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" podUID="705ab665-d772-4ba3-af1a-56885b1f1ff7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:48:08.185686 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:08.185564 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-841de3b011bd: secret "agc-cred-841de3b011bd" not found Nov 01 06:48:08.185686 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:08.185600 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-97c3302b7889: secret "agc-cred-97c3302b7889" not found Nov 01 06:48:08.185686 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:08.185614 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-c3e56f59453b: secret "agc-cred-c3e56f59453b" not found Nov 01 06:48:08.185686 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:08.185627 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc: [secret "agc-cred-841de3b011bd" not found, secret "agc-cred-97c3302b7889" not found, secret "agc-cred-c3e56f59453b" not found] Nov 01 06:48:08.186165 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:08.186140 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-provider-credentials podName:e5a2a7f9-63f8-44ae-a226-c5fd9574643f nodeName:}" failed. No retries permitted until 2026-11-01 06:48:16.186120251 +0000 UTC m=+2305.949920099 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" (UID: "e5a2a7f9-63f8-44ae-a226-c5fd9574643f") : [secret "agc-cred-841de3b011bd" not found, secret "agc-cred-97c3302b7889" not found, secret "agc-cred-c3e56f59453b" not found] Nov 01 06:48:08.392460 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:08.392419 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" event={"ID":"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7","Type":"ContainerStarted","Data":"cc3e5848179574477e3a2efa31997d53ef938d3d5ff98c181ae7ad80023a6cf3"} Nov 01 06:48:08.392870 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:08.392607 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" Nov 01 06:48:08.509956 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:08.509854 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" podStartSLOduration=2.653998453 podStartE2EDuration="3.509825672s" totalImagesPullingTime="855.827219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:48:08.506387019 +0000 UTC m=+2298.270186877" watchObservedRunningTime="2026-11-01 06:48:08.509825672 +0000 UTC m=+2298.273625529" Nov 01 06:48:10.097859 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:10.097807 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh"] Nov 01 06:48:10.099122 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:10.099097 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" podUID="705ab665-d772-4ba3-af1a-56885b1f1ff7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:48:10.324782 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:10.324740 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" podUID="ec39162f-e923-4d1f-b491-529ba0b08a1c" containerName="payload-processing" containerID="cri-o://70546a6a52d6094987b06e9069b8fb9341db1d28f79c1fbcd286f381bee2e985" gracePeriod=25 Nov 01 06:48:10.591144 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:10.591117 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" Nov 01 06:48:10.614913 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:10.614882 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec39162f-e923-4d1f-b491-529ba0b08a1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec39162f-e923-4d1f-b491-529ba0b08a1c-plugins-config-volume\") pod \"ec39162f-e923-4d1f-b491-529ba0b08a1c\" (UID: \"ec39162f-e923-4d1f-b491-529ba0b08a1c\") " Nov 01 06:48:10.614913 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:10.614920 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec39162f-e923-4d1f-b491-529ba0b08a1c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec39162f-e923-4d1f-b491-529ba0b08a1c-provider-credentials\") pod \"ec39162f-e923-4d1f-b491-529ba0b08a1c\" (UID: \"ec39162f-e923-4d1f-b491-529ba0b08a1c\") " Nov 01 06:48:10.615180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:10.614961 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec39162f-e923-4d1f-b491-529ba0b08a1c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec39162f-e923-4d1f-b491-529ba0b08a1c-routing-overlay-volume\") pod \"ec39162f-e923-4d1f-b491-529ba0b08a1c\" (UID: \"ec39162f-e923-4d1f-b491-529ba0b08a1c\") " Nov 01 06:48:10.615180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:10.615022 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec39162f-e923-4d1f-b491-529ba0b08a1c-kube-api-access-blwp7\" (UniqueName: \"kubernetes.io/projected/ec39162f-e923-4d1f-b491-529ba0b08a1c-kube-api-access-blwp7\") pod \"ec39162f-e923-4d1f-b491-529ba0b08a1c\" (UID: \"ec39162f-e923-4d1f-b491-529ba0b08a1c\") " Nov 01 06:48:10.615381 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:10.615352 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec39162f-e923-4d1f-b491-529ba0b08a1c-plugins-config-volume" pod "ec39162f-e923-4d1f-b491-529ba0b08a1c" (UID: "ec39162f-e923-4d1f-b491-529ba0b08a1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:10.615381 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:10.615365 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec39162f-e923-4d1f-b491-529ba0b08a1c-routing-overlay-volume" pod "ec39162f-e923-4d1f-b491-529ba0b08a1c" (UID: "ec39162f-e923-4d1f-b491-529ba0b08a1c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:10.615581 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:10.615506 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec39162f-e923-4d1f-b491-529ba0b08a1c-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:10.615581 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:10.615532 4385 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec39162f-e923-4d1f-b491-529ba0b08a1c-routing-overlay-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:10.617457 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:10.617433 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec39162f-e923-4d1f-b491-529ba0b08a1c-provider-credentials" pod "ec39162f-e923-4d1f-b491-529ba0b08a1c" (UID: "ec39162f-e923-4d1f-b491-529ba0b08a1c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:10.617572 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:10.617460 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec39162f-e923-4d1f-b491-529ba0b08a1c-kube-api-access-blwp7" pod "ec39162f-e923-4d1f-b491-529ba0b08a1c" (UID: "ec39162f-e923-4d1f-b491-529ba0b08a1c"). InnerVolumeSpecName "kube-api-access-blwp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:10.716302 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:10.716227 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blwp7\" (UniqueName: \"kubernetes.io/projected/ec39162f-e923-4d1f-b491-529ba0b08a1c-kube-api-access-blwp7\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:10.716302 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:10.716255 4385 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec39162f-e923-4d1f-b491-529ba0b08a1c-provider-credentials\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:10.734334 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:10.734288 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:11.414672 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:11.414636 4385 generic.go:356] "Generic (PLEG): container finished" podID="ec39162f-e923-4d1f-b491-529ba0b08a1c" containerID="70546a6a52d6094987b06e9069b8fb9341db1d28f79c1fbcd286f381bee2e985" exitCode=0 Nov 01 06:48:11.415215 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:11.414690 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" Nov 01 06:48:11.415215 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:11.414719 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" event={"ID":"ec39162f-e923-4d1f-b491-529ba0b08a1c","Type":"ContainerDied","Data":"70546a6a52d6094987b06e9069b8fb9341db1d28f79c1fbcd286f381bee2e985"} Nov 01 06:48:11.415215 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:11.414781 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl" event={"ID":"ec39162f-e923-4d1f-b491-529ba0b08a1c","Type":"ContainerDied","Data":"89f3b25b0cc6ef9b061e6d159a055d5e24ddece3b2efd4ce24d50923606402a2"} Nov 01 06:48:11.415215 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:11.414810 4385 scope.go:122] "RemoveContainer" containerID="70546a6a52d6094987b06e9069b8fb9341db1d28f79c1fbcd286f381bee2e985" Nov 01 06:48:11.427676 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:11.427659 4385 scope.go:122] "RemoveContainer" containerID="70546a6a52d6094987b06e9069b8fb9341db1d28f79c1fbcd286f381bee2e985" Nov 01 06:48:11.428021 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:11.427997 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70546a6a52d6094987b06e9069b8fb9341db1d28f79c1fbcd286f381bee2e985\": container with ID starting with 70546a6a52d6094987b06e9069b8fb9341db1d28f79c1fbcd286f381bee2e985 not found: ID does not exist" containerID="70546a6a52d6094987b06e9069b8fb9341db1d28f79c1fbcd286f381bee2e985" Nov 01 06:48:11.428085 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:11.428038 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70546a6a52d6094987b06e9069b8fb9341db1d28f79c1fbcd286f381bee2e985"} err="failed to get container status \"70546a6a52d6094987b06e9069b8fb9341db1d28f79c1fbcd286f381bee2e985\": rpc error: code = NotFound desc = could not find container \"70546a6a52d6094987b06e9069b8fb9341db1d28f79c1fbcd286f381bee2e985\": container with ID starting with 70546a6a52d6094987b06e9069b8fb9341db1d28f79c1fbcd286f381bee2e985 not found: ID does not exist" Nov 01 06:48:11.578296 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:11.578257 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl"] Nov 01 06:48:11.581649 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:11.581618 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7cgstfl"] Nov 01 06:48:12.385034 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:12.385004 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec39162f-e923-4d1f-b491-529ba0b08a1c" path="/var/lib/kubelet/pods/ec39162f-e923-4d1f-b491-529ba0b08a1c/volumes" Nov 01 06:48:14.405222 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:14.405197 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" Nov 01 06:48:15.098801 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.098758 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" podUID="705ab665-d772-4ba3-af1a-56885b1f1ff7" containerName="payload-processing" containerID="cri-o://a2ce75c1f580ebcf29e05979bb3a566745c02ca156cbda2c40f316d2bbbe1b68" gracePeriod=25 Nov 01 06:48:15.263454 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:15.263421 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-25719d/agc-cred-c7ba2136e663: secret "agc-cred-c7ba2136e663" not found Nov 01 06:48:15.263454 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:15.263454 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-25719d/agc-cred-d41a0e39961b: secret "agc-cred-d41a0e39961b" not found Nov 01 06:48:15.263454 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:15.263461 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-25719d/agc-cred-f6f2772a70c5: secret "agc-cred-f6f2772a70c5" not found Nov 01 06:48:15.263454 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:15.263474 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk: [secret "agc-cred-c7ba2136e663" not found, secret "agc-cred-d41a0e39961b" not found, secret "agc-cred-f6f2772a70c5" not found] Nov 01 06:48:15.264246 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:15.263531 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-provider-credentials podName:46bcb2e8-bdb2-4632-911d-d0668846115a nodeName:}" failed. No retries permitted until 2026-11-01 06:48:31.263516434 +0000 UTC m=+2321.027316269 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" (UID: "46bcb2e8-bdb2-4632-911d-d0668846115a") : [secret "agc-cred-c7ba2136e663" not found, secret "agc-cred-d41a0e39961b" not found, secret "agc-cred-f6f2772a70c5" not found] Nov 01 06:48:15.422090 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.422056 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" Nov 01 06:48:15.441124 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.441086 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:15.452708 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.452672 4385 generic.go:356] "Generic (PLEG): container finished" podID="705ab665-d772-4ba3-af1a-56885b1f1ff7" containerID="a2ce75c1f580ebcf29e05979bb3a566745c02ca156cbda2c40f316d2bbbe1b68" exitCode=0 Nov 01 06:48:15.452885 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.452713 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" Nov 01 06:48:15.452885 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.452761 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" event={"ID":"705ab665-d772-4ba3-af1a-56885b1f1ff7","Type":"ContainerDied","Data":"a2ce75c1f580ebcf29e05979bb3a566745c02ca156cbda2c40f316d2bbbe1b68"} Nov 01 06:48:15.452885 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.452870 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh" event={"ID":"705ab665-d772-4ba3-af1a-56885b1f1ff7","Type":"ContainerDied","Data":"4d16eb761a1c88e99eed7335d6e8fbbadaff3699ac09ac65bf5f3656ca50838c"} Nov 01 06:48:15.452996 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.452901 4385 scope.go:122] "RemoveContainer" containerID="a2ce75c1f580ebcf29e05979bb3a566745c02ca156cbda2c40f316d2bbbe1b68" Nov 01 06:48:15.466327 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.466309 4385 scope.go:122] "RemoveContainer" containerID="a2ce75c1f580ebcf29e05979bb3a566745c02ca156cbda2c40f316d2bbbe1b68" Nov 01 06:48:15.466611 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:15.466593 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ce75c1f580ebcf29e05979bb3a566745c02ca156cbda2c40f316d2bbbe1b68\": container with ID starting with a2ce75c1f580ebcf29e05979bb3a566745c02ca156cbda2c40f316d2bbbe1b68 not found: ID does not exist" containerID="a2ce75c1f580ebcf29e05979bb3a566745c02ca156cbda2c40f316d2bbbe1b68" Nov 01 06:48:15.466657 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.466620 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ce75c1f580ebcf29e05979bb3a566745c02ca156cbda2c40f316d2bbbe1b68"} err="failed to get container status \"a2ce75c1f580ebcf29e05979bb3a566745c02ca156cbda2c40f316d2bbbe1b68\": rpc error: code = NotFound desc = could not find container \"a2ce75c1f580ebcf29e05979bb3a566745c02ca156cbda2c40f316d2bbbe1b68\": container with ID starting with a2ce75c1f580ebcf29e05979bb3a566745c02ca156cbda2c40f316d2bbbe1b68 not found: ID does not exist" Nov 01 06:48:15.490881 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.490860 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/705ab665-d772-4ba3-af1a-56885b1f1ff7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/705ab665-d772-4ba3-af1a-56885b1f1ff7-provider-credentials\") pod \"705ab665-d772-4ba3-af1a-56885b1f1ff7\" (UID: \"705ab665-d772-4ba3-af1a-56885b1f1ff7\") " Nov 01 06:48:15.491039 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.490969 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/705ab665-d772-4ba3-af1a-56885b1f1ff7-kube-api-access-lrllt\" (UniqueName: \"kubernetes.io/projected/705ab665-d772-4ba3-af1a-56885b1f1ff7-kube-api-access-lrllt\") pod \"705ab665-d772-4ba3-af1a-56885b1f1ff7\" (UID: \"705ab665-d772-4ba3-af1a-56885b1f1ff7\") " Nov 01 06:48:15.491039 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.491014 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/705ab665-d772-4ba3-af1a-56885b1f1ff7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/705ab665-d772-4ba3-af1a-56885b1f1ff7-routing-overlay-volume\") pod \"705ab665-d772-4ba3-af1a-56885b1f1ff7\" (UID: \"705ab665-d772-4ba3-af1a-56885b1f1ff7\") " Nov 01 06:48:15.491163 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.491055 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/705ab665-d772-4ba3-af1a-56885b1f1ff7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705ab665-d772-4ba3-af1a-56885b1f1ff7-plugins-config-volume\") pod \"705ab665-d772-4ba3-af1a-56885b1f1ff7\" (UID: \"705ab665-d772-4ba3-af1a-56885b1f1ff7\") " Nov 01 06:48:15.491448 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.491426 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/705ab665-d772-4ba3-af1a-56885b1f1ff7-routing-overlay-volume" pod "705ab665-d772-4ba3-af1a-56885b1f1ff7" (UID: "705ab665-d772-4ba3-af1a-56885b1f1ff7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:15.491574 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.491548 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/705ab665-d772-4ba3-af1a-56885b1f1ff7-plugins-config-volume" pod "705ab665-d772-4ba3-af1a-56885b1f1ff7" (UID: "705ab665-d772-4ba3-af1a-56885b1f1ff7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:15.491643 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.491628 4385 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/705ab665-d772-4ba3-af1a-56885b1f1ff7-routing-overlay-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:15.491701 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.491649 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705ab665-d772-4ba3-af1a-56885b1f1ff7-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:15.493873 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.493850 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705ab665-d772-4ba3-af1a-56885b1f1ff7-provider-credentials" pod "705ab665-d772-4ba3-af1a-56885b1f1ff7" (UID: "705ab665-d772-4ba3-af1a-56885b1f1ff7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:15.493987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.493872 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705ab665-d772-4ba3-af1a-56885b1f1ff7-kube-api-access-lrllt" pod "705ab665-d772-4ba3-af1a-56885b1f1ff7" (UID: "705ab665-d772-4ba3-af1a-56885b1f1ff7"). InnerVolumeSpecName "kube-api-access-lrllt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:15.589893 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.589305 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:15.592798 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.592765 4385 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/705ab665-d772-4ba3-af1a-56885b1f1ff7-provider-credentials\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:15.592798 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.592799 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrllt\" (UniqueName: \"kubernetes.io/projected/705ab665-d772-4ba3-af1a-56885b1f1ff7-kube-api-access-lrllt\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:15.864934 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.864895 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:15.874190 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.874151 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh"] Nov 01 06:48:15.878609 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:15.878582 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-54d5lv8jh"] Nov 01 06:48:16.200516 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:16.200487 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-841de3b011bd: secret "agc-cred-841de3b011bd" not found Nov 01 06:48:16.200516 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:16.200508 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-97c3302b7889: secret "agc-cred-97c3302b7889" not found Nov 01 06:48:16.200516 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:16.200514 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-c3e56f59453b: secret "agc-cred-c3e56f59453b" not found Nov 01 06:48:16.200516 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:16.200525 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc: [secret "agc-cred-841de3b011bd" not found, secret "agc-cred-97c3302b7889" not found, secret "agc-cred-c3e56f59453b" not found] Nov 01 06:48:16.200791 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:16.200580 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-provider-credentials podName:e5a2a7f9-63f8-44ae-a226-c5fd9574643f nodeName:}" failed. No retries permitted until 2026-11-01 06:48:32.200565152 +0000 UTC m=+2321.964364988 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" (UID: "e5a2a7f9-63f8-44ae-a226-c5fd9574643f") : [secret "agc-cred-841de3b011bd" not found, secret "agc-cred-97c3302b7889" not found, secret "agc-cred-c3e56f59453b" not found] Nov 01 06:48:16.386618 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:16.386582 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="705ab665-d772-4ba3-af1a-56885b1f1ff7" path="/var/lib/kubelet/pods/705ab665-d772-4ba3-af1a-56885b1f1ff7/volumes" Nov 01 06:48:19.721343 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:19.721309 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:20.253244 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:20.253195 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:24.398606 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:24.398573 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:24.547554 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:24.547518 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:25.007727 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:25.007691 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:27.405954 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:27.405906 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc"] Nov 01 06:48:27.546420 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:27.546382 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:27.738250 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:27.738140 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:28.292609 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:28.292575 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:28.553902 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:28.550932 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:28.691151 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:28.691106 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:28.980183 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:28.980149 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:29.284436 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:29.284359 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:29.729636 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:29.729598 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:30.083429 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:30.083402 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk"] Nov 01 06:48:30.252786 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:30.252746 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:31.291649 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:31.291618 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-25719d/agc-cred-294b20a2cf15: secret "agc-cred-294b20a2cf15" not found Nov 01 06:48:31.291649 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:31.291644 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-25719d/agc-cred-c7ba2136e663: secret "agc-cred-c7ba2136e663" not found Nov 01 06:48:31.291649 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:31.291650 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-25719d/agc-cred-d41a0e39961b: secret "agc-cred-d41a0e39961b" not found Nov 01 06:48:31.291649 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:31.291656 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-25719d/agc-cred-f6f2772a70c5: secret "agc-cred-f6f2772a70c5" not found Nov 01 06:48:31.292153 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:31.291665 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk: [secret "agc-cred-294b20a2cf15" not found, secret "agc-cred-c7ba2136e663" not found, secret "agc-cred-d41a0e39961b" not found, secret "agc-cred-f6f2772a70c5" not found] Nov 01 06:48:31.292153 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:31.291724 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-provider-credentials podName:46bcb2e8-bdb2-4632-911d-d0668846115a nodeName:}" failed. No retries permitted until 2026-11-01 06:49:03.291710572 +0000 UTC m=+2353.055510406 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" (UID: "46bcb2e8-bdb2-4632-911d-d0668846115a") : [secret "agc-cred-294b20a2cf15" not found, secret "agc-cred-c7ba2136e663" not found, secret "agc-cred-d41a0e39961b" not found, secret "agc-cred-f6f2772a70c5" not found] Nov 01 06:48:32.301954 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:32.301928 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-529628bc0fc6: secret "agc-cred-529628bc0fc6" not found Nov 01 06:48:32.301954 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:32.301950 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-841de3b011bd: secret "agc-cred-841de3b011bd" not found Nov 01 06:48:32.301954 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:32.301957 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-97c3302b7889: secret "agc-cred-97c3302b7889" not found Nov 01 06:48:32.301954 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:32.301964 4385 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-c3e56f59453b: secret "agc-cred-c3e56f59453b" not found Nov 01 06:48:32.302446 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:32.301973 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc: [secret "agc-cred-529628bc0fc6" not found, secret "agc-cred-841de3b011bd" not found, secret "agc-cred-97c3302b7889" not found, secret "agc-cred-c3e56f59453b" not found] Nov 01 06:48:32.302446 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:32.302042 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-provider-credentials podName:e5a2a7f9-63f8-44ae-a226-c5fd9574643f nodeName:}" failed. No retries permitted until 2026-11-01 06:49:04.302020906 +0000 UTC m=+2354.065820760 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" (UID: "e5a2a7f9-63f8-44ae-a226-c5fd9574643f") : [secret "agc-cred-529628bc0fc6" not found, secret "agc-cred-841de3b011bd" not found, secret "agc-cred-97c3302b7889" not found, secret "agc-cred-c3e56f59453b" not found] Nov 01 06:48:32.407003 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:32.406963 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" podUID="e5a2a7f9-63f8-44ae-a226-c5fd9574643f" containerName="payload-processing" containerID="cri-o://dc1def466425333fe816195caba301eb35ced9314c1307d5f0311bdae5c2c61e" gracePeriod=25 Nov 01 06:48:32.588337 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:32.588079 4385 generic.go:356] "Generic (PLEG): container finished" podID="e5a2a7f9-63f8-44ae-a226-c5fd9574643f" containerID="dc1def466425333fe816195caba301eb35ced9314c1307d5f0311bdae5c2c61e" exitCode=0 Nov 01 06:48:32.588337 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:32.588282 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" event={"ID":"e5a2a7f9-63f8-44ae-a226-c5fd9574643f","Type":"ContainerDied","Data":"dc1def466425333fe816195caba301eb35ced9314c1307d5f0311bdae5c2c61e"} Nov 01 06:48:32.682752 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:32.682727 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" Nov 01 06:48:32.806665 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:32.806633 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-plugins-config-volume\") pod \"e5a2a7f9-63f8-44ae-a226-c5fd9574643f\" (UID: \"e5a2a7f9-63f8-44ae-a226-c5fd9574643f\") " Nov 01 06:48:32.806861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:32.806802 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-kube-api-access-tcvmt\" (UniqueName: \"kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-kube-api-access-tcvmt\") pod \"e5a2a7f9-63f8-44ae-a226-c5fd9574643f\" (UID: \"e5a2a7f9-63f8-44ae-a226-c5fd9574643f\") " Nov 01 06:48:32.806923 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:32.806879 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-provider-credentials\") pod \"e5a2a7f9-63f8-44ae-a226-c5fd9574643f\" (UID: \"e5a2a7f9-63f8-44ae-a226-c5fd9574643f\") " Nov 01 06:48:32.806923 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:32.806915 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-routing-overlay-volume\") pod \"e5a2a7f9-63f8-44ae-a226-c5fd9574643f\" (UID: \"e5a2a7f9-63f8-44ae-a226-c5fd9574643f\") " Nov 01 06:48:32.807074 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:32.807046 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-plugins-config-volume" pod "e5a2a7f9-63f8-44ae-a226-c5fd9574643f" (UID: "e5a2a7f9-63f8-44ae-a226-c5fd9574643f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:32.807313 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:32.807292 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-routing-overlay-volume" pod "e5a2a7f9-63f8-44ae-a226-c5fd9574643f" (UID: "e5a2a7f9-63f8-44ae-a226-c5fd9574643f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:32.807440 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:32.807422 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:32.807479 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:32.807448 4385 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-routing-overlay-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:32.809232 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:32.809204 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-kube-api-access-tcvmt" pod "e5a2a7f9-63f8-44ae-a226-c5fd9574643f" (UID: "e5a2a7f9-63f8-44ae-a226-c5fd9574643f"). InnerVolumeSpecName "kube-api-access-tcvmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:32.809338 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:32.809260 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-provider-credentials" pod "e5a2a7f9-63f8-44ae-a226-c5fd9574643f" (UID: "e5a2a7f9-63f8-44ae-a226-c5fd9574643f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:32.908729 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:32.908655 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcvmt\" (UniqueName: \"kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-kube-api-access-tcvmt\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:32.908729 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:32.908683 4385 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5a2a7f9-63f8-44ae-a226-c5fd9574643f-provider-credentials\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:32.945606 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:32.945566 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:33.606866 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:33.604247 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" event={"ID":"e5a2a7f9-63f8-44ae-a226-c5fd9574643f","Type":"ContainerDied","Data":"c1a18ab2131b6a54e8e0b74c9b3914825d0735a82d39e05f3272f5173f51e328"} Nov 01 06:48:33.606866 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:33.604314 4385 scope.go:122] "RemoveContainer" containerID="dc1def466425333fe816195caba301eb35ced9314c1307d5f0311bdae5c2c61e" Nov 01 06:48:33.606866 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:33.604469 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc" Nov 01 06:48:33.786077 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:33.786043 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc"] Nov 01 06:48:33.834341 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:33.834312 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-5d96c85hc"] Nov 01 06:48:34.108671 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:34.108627 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:34.262616 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:34.262582 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:34.384275 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:34.384202 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5a2a7f9-63f8-44ae-a226-c5fd9574643f" path="/var/lib/kubelet/pods/e5a2a7f9-63f8-44ae-a226-c5fd9574643f/volumes" Nov 01 06:48:34.531258 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:34.531220 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:34.991333 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:34.991287 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:35.084604 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.084563 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" podUID="46bcb2e8-bdb2-4632-911d-d0668846115a" containerName="payload-processing" containerID="cri-o://306a397e97f18fc4d7442a0b6fc1507014fcfab883495394435f0999b87cd006" gracePeriod=25 Nov 01 06:48:35.157107 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.157068 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:35.355379 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.355354 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" Nov 01 06:48:35.442539 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.442503 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-provider-credentials\") pod \"46bcb2e8-bdb2-4632-911d-d0668846115a\" (UID: \"46bcb2e8-bdb2-4632-911d-d0668846115a\") " Nov 01 06:48:35.442733 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.442572 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46bcb2e8-bdb2-4632-911d-d0668846115a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46bcb2e8-bdb2-4632-911d-d0668846115a-plugins-config-volume\") pod \"46bcb2e8-bdb2-4632-911d-d0668846115a\" (UID: \"46bcb2e8-bdb2-4632-911d-d0668846115a\") " Nov 01 06:48:35.442733 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.442601 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-kube-api-access-p72dx\" (UniqueName: \"kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-kube-api-access-p72dx\") pod \"46bcb2e8-bdb2-4632-911d-d0668846115a\" (UID: \"46bcb2e8-bdb2-4632-911d-d0668846115a\") " Nov 01 06:48:35.442878 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.442730 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46bcb2e8-bdb2-4632-911d-d0668846115a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46bcb2e8-bdb2-4632-911d-d0668846115a-routing-overlay-volume\") pod \"46bcb2e8-bdb2-4632-911d-d0668846115a\" (UID: \"46bcb2e8-bdb2-4632-911d-d0668846115a\") " Nov 01 06:48:35.442960 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.442937 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46bcb2e8-bdb2-4632-911d-d0668846115a-plugins-config-volume" pod "46bcb2e8-bdb2-4632-911d-d0668846115a" (UID: "46bcb2e8-bdb2-4632-911d-d0668846115a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:35.443144 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.443111 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46bcb2e8-bdb2-4632-911d-d0668846115a-routing-overlay-volume" pod "46bcb2e8-bdb2-4632-911d-d0668846115a" (UID: "46bcb2e8-bdb2-4632-911d-d0668846115a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:35.443457 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.443434 4385 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/46bcb2e8-bdb2-4632-911d-d0668846115a-routing-overlay-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:35.443557 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.443464 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46bcb2e8-bdb2-4632-911d-d0668846115a-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:35.445000 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.444977 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-provider-credentials" pod "46bcb2e8-bdb2-4632-911d-d0668846115a" (UID: "46bcb2e8-bdb2-4632-911d-d0668846115a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:35.445082 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.444980 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-kube-api-access-p72dx" pod "46bcb2e8-bdb2-4632-911d-d0668846115a" (UID: "46bcb2e8-bdb2-4632-911d-d0668846115a"). InnerVolumeSpecName "kube-api-access-p72dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:35.544698 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.544621 4385 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-provider-credentials\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:35.544698 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.544660 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p72dx\" (UniqueName: \"kubernetes.io/projected/46bcb2e8-bdb2-4632-911d-d0668846115a-kube-api-access-p72dx\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:35.619797 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.619758 4385 generic.go:356] "Generic (PLEG): container finished" podID="46bcb2e8-bdb2-4632-911d-d0668846115a" containerID="306a397e97f18fc4d7442a0b6fc1507014fcfab883495394435f0999b87cd006" exitCode=0 Nov 01 06:48:35.620024 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.619811 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" Nov 01 06:48:35.620024 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.619815 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" event={"ID":"46bcb2e8-bdb2-4632-911d-d0668846115a","Type":"ContainerDied","Data":"306a397e97f18fc4d7442a0b6fc1507014fcfab883495394435f0999b87cd006"} Nov 01 06:48:35.620024 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.619943 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk" event={"ID":"46bcb2e8-bdb2-4632-911d-d0668846115a","Type":"ContainerDied","Data":"e037ba3bab900596a68e3b643dd4be82d41f0cf218e0ae69a6d3cbe4bf445c60"} Nov 01 06:48:35.620024 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.619988 4385 scope.go:122] "RemoveContainer" containerID="306a397e97f18fc4d7442a0b6fc1507014fcfab883495394435f0999b87cd006" Nov 01 06:48:35.631386 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.631370 4385 scope.go:122] "RemoveContainer" containerID="306a397e97f18fc4d7442a0b6fc1507014fcfab883495394435f0999b87cd006" Nov 01 06:48:35.631667 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:35.631648 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"306a397e97f18fc4d7442a0b6fc1507014fcfab883495394435f0999b87cd006\": container with ID starting with 306a397e97f18fc4d7442a0b6fc1507014fcfab883495394435f0999b87cd006 not found: ID does not exist" containerID="306a397e97f18fc4d7442a0b6fc1507014fcfab883495394435f0999b87cd006" Nov 01 06:48:35.631721 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.631674 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306a397e97f18fc4d7442a0b6fc1507014fcfab883495394435f0999b87cd006"} err="failed to get container status \"306a397e97f18fc4d7442a0b6fc1507014fcfab883495394435f0999b87cd006\": rpc error: code = NotFound desc = could not find container \"306a397e97f18fc4d7442a0b6fc1507014fcfab883495394435f0999b87cd006\": container with ID starting with 306a397e97f18fc4d7442a0b6fc1507014fcfab883495394435f0999b87cd006 not found: ID does not exist" Nov 01 06:48:35.784550 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.784511 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk"] Nov 01 06:48:35.787940 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:35.787920 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75ddv5kdk"] Nov 01 06:48:36.384214 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:36.384182 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46bcb2e8-bdb2-4632-911d-d0668846115a" path="/var/lib/kubelet/pods/46bcb2e8-bdb2-4632-911d-d0668846115a/volumes" Nov 01 06:48:36.780007 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:48:36.779923 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:48:38.918639 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:38.918602 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:39.626066 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:39.626022 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:40.220385 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:40.220341 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:40.491749 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:40.491659 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:40.656656 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:40.656618 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:40.814588 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:40.814502 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:41.012426 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:41.012391 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:41.396860 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:41.388006 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:41.549699 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:41.549669 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:44.458784 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:44.458747 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:45.095724 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:45.095682 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:45.379506 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:45.379370 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:45.910393 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:45.910363 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:46.983436 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:46.983396 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:50.006164 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:50.006120 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:50.823851 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:50.823806 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:51.029240 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.029201 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:51.218474 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.218396 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt"] Nov 01 06:48:51.219205 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.219185 4385 cpu_manager.go:414] "containerMap: removing container" podUID="705ab665-d772-4ba3-af1a-56885b1f1ff7" containerName="payload-processing" Nov 01 06:48:51.219287 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.219208 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705ab665-d772-4ba3-af1a-56885b1f1ff7" containerName="payload-processing" Nov 01 06:48:51.219287 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.219227 4385 cpu_manager.go:414] "containerMap: removing container" podUID="e5a2a7f9-63f8-44ae-a226-c5fd9574643f" containerName="payload-processing" Nov 01 06:48:51.219287 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.219234 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5a2a7f9-63f8-44ae-a226-c5fd9574643f" containerName="payload-processing" Nov 01 06:48:51.219287 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.219259 4385 cpu_manager.go:414] "containerMap: removing container" podUID="ec39162f-e923-4d1f-b491-529ba0b08a1c" containerName="payload-processing" Nov 01 06:48:51.219287 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.219267 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec39162f-e923-4d1f-b491-529ba0b08a1c" containerName="payload-processing" Nov 01 06:48:51.219487 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.219295 4385 cpu_manager.go:414] "containerMap: removing container" podUID="46bcb2e8-bdb2-4632-911d-d0668846115a" containerName="payload-processing" Nov 01 06:48:51.219487 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.219302 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46bcb2e8-bdb2-4632-911d-d0668846115a" containerName="payload-processing" Nov 01 06:48:51.219487 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.219427 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5a2a7f9-63f8-44ae-a226-c5fd9574643f" containerName="payload-processing" Nov 01 06:48:51.219487 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.219452 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec39162f-e923-4d1f-b491-529ba0b08a1c" containerName="payload-processing" Nov 01 06:48:51.219487 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.219464 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="46bcb2e8-bdb2-4632-911d-d0668846115a" containerName="payload-processing" Nov 01 06:48:51.219487 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.219477 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="705ab665-d772-4ba3-af1a-56885b1f1ff7" containerName="payload-processing" Nov 01 06:48:51.222902 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.222884 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt" Nov 01 06:48:51.259256 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.259220 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wp4c\" (UniqueName: \"kubernetes.io/projected/94a78d83-1b31-4176-95eb-b3b03b076419-kube-api-access-6wp4c\") pod \"payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt\" (UID: \"94a78d83-1b31-4176-95eb-b3b03b076419\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt" Nov 01 06:48:51.259398 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.259271 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a78d83-1b31-4176-95eb-b3b03b076419-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt\" (UID: \"94a78d83-1b31-4176-95eb-b3b03b076419\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt" Nov 01 06:48:51.278097 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.278064 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt"] Nov 01 06:48:51.336859 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.335249 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-581487\"" Nov 01 06:48:51.336859 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.335430 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-581487-dockercfg-22gf8\"" Nov 01 06:48:51.364497 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.364447 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wp4c\" (UniqueName: \"kubernetes.io/projected/94a78d83-1b31-4176-95eb-b3b03b076419-kube-api-access-6wp4c\") pod \"payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt\" (UID: \"94a78d83-1b31-4176-95eb-b3b03b076419\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt" Nov 01 06:48:51.364782 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.364760 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a78d83-1b31-4176-95eb-b3b03b076419-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt\" (UID: \"94a78d83-1b31-4176-95eb-b3b03b076419\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt" Nov 01 06:48:51.365904 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.365880 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a78d83-1b31-4176-95eb-b3b03b076419-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt\" (UID: \"94a78d83-1b31-4176-95eb-b3b03b076419\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt" Nov 01 06:48:51.421794 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.421768 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wp4c\" (UniqueName: \"kubernetes.io/projected/94a78d83-1b31-4176-95eb-b3b03b076419-kube-api-access-6wp4c\") pod \"payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt\" (UID: \"94a78d83-1b31-4176-95eb-b3b03b076419\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt" Nov 01 06:48:51.537110 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.537020 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt" Nov 01 06:48:51.921580 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:51.921537 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt"] Nov 01 06:48:51.922040 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:48:51.922006 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a78d83_1b31_4176_95eb_b3b03b076419.slice/crio-9dbbd0963b33a9e54e24852ebf64e323c9f3896b0ca80fd3931b4632b6db48ed WatchSource:0}: Error finding container 9dbbd0963b33a9e54e24852ebf64e323c9f3896b0ca80fd3931b4632b6db48ed: Status 404 returned error can't find the container with id 9dbbd0963b33a9e54e24852ebf64e323c9f3896b0ca80fd3931b4632b6db48ed Nov 01 06:48:52.747281 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:52.747247 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt" event={"ID":"94a78d83-1b31-4176-95eb-b3b03b076419","Type":"ContainerStarted","Data":"bf3bc0a112dd2bb69eec360fb7bcc657271fab05cbdc42ccfc1ddd5c1e02eab4"} Nov 01 06:48:52.747695 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:52.747308 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt" event={"ID":"94a78d83-1b31-4176-95eb-b3b03b076419","Type":"ContainerStarted","Data":"9dbbd0963b33a9e54e24852ebf64e323c9f3896b0ca80fd3931b4632b6db48ed"} Nov 01 06:48:52.747695 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:52.747424 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt" Nov 01 06:48:52.864247 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:52.864173 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt" podStartSLOduration=1.864154831 podStartE2EDuration="1.864154831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:48:52.860113603 +0000 UTC m=+2342.623913463" watchObservedRunningTime="2026-11-01 06:48:52.864154831 +0000 UTC m=+2342.627954689" Nov 01 06:48:54.698441 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:54.698403 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:48:59.656086 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:48:59.656040 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:03.757787 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:03.757757 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt" Nov 01 06:49:04.292967 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:04.292927 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:06.779293 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:49:06.779253 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:49:09.244860 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:09.244702 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:14.215993 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:14.215952 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:18.382338 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:18.382291 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:18.935453 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:18.935426 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:20.328610 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:20.328574 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:21.698052 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:21.698011 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:24.230906 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:24.230873 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:24.421442 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:24.421412 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:25.613448 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:25.613415 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:26.397693 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:26.397654 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:26.543870 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:26.543816 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:29.504179 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:29.504091 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:29.781352 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:29.781183 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:32.027815 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:32.027726 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:32.444422 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:32.444377 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:32.594413 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:32.594370 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:35.192321 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:35.192280 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:36.780308 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:49:36.780270 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:49:37.426183 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:37.426144 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:39.832304 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:39.832250 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:44.462589 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:44.462550 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:47.296045 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:47.296003 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:49.107413 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:49.107364 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:53.659862 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:53.659807 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:53.815067 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:53.815016 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:53.960608 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:53.960508 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:57.934332 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:57.934292 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:49:59.633524 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:49:59.633480 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:01.162291 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:01.162253 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:01.372972 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:01.372935 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:02.634683 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:02.634645 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:03.133824 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:03.133786 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:03.601339 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:03.601307 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:03.795790 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:03.795758 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:04.095630 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:04.095596 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:04.725797 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:04.725747 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:06.780344 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:50:06.780304 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:50:09.447563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:09.447529 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:10.263613 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:10.263582 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:10.461560 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:10.461527 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:10.828627 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:10.828584 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:13.606612 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:13.606571 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:15.021286 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:15.021252 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:15.175460 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:15.175420 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:15.365550 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:15.365520 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:16.548993 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:16.548952 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:16.762087 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:16.762060 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:16.919781 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:16.919747 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:17.076872 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:17.076813 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:17.613200 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:17.613159 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:17.760599 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:17.760560 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:17.916560 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:17.916467 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:20.618905 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:20.618866 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:20.811670 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:20.811630 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:23.250657 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:23.250622 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:23.310664 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:23.310622 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j"] Nov 01 06:50:23.678715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:23.678677 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:23.893857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:23.889646 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:24.056693 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:24.056594 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:25.623984 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:25.623953 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:28.070535 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:28.070497 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:28.268257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:28.267941 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:28.314989 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:28.314939 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j" podUID="91f7307b-e9c5-45ee-871c-d1de04507c0a" containerName="payload-processing" containerID="cri-o://d614f9ac349a5f1c1ead58a1f9ab330f50da70759400c9138f5ef3ce6aaaa40f" gracePeriod=25 Nov 01 06:50:28.551367 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:28.551314 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:28.971882 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:28.971854 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j" Nov 01 06:50:28.994360 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:28.994319 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91f7307b-e9c5-45ee-871c-d1de04507c0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91f7307b-e9c5-45ee-871c-d1de04507c0a-plugins-config-volume\") pod \"91f7307b-e9c5-45ee-871c-d1de04507c0a\" (UID: \"91f7307b-e9c5-45ee-871c-d1de04507c0a\") " Nov 01 06:50:28.994588 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:28.994547 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f7307b-e9c5-45ee-871c-d1de04507c0a-kube-api-access-t68jg\" (UniqueName: \"kubernetes.io/projected/91f7307b-e9c5-45ee-871c-d1de04507c0a-kube-api-access-t68jg\") pod \"91f7307b-e9c5-45ee-871c-d1de04507c0a\" (UID: \"91f7307b-e9c5-45ee-871c-d1de04507c0a\") " Nov 01 06:50:28.995142 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:28.995116 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f7307b-e9c5-45ee-871c-d1de04507c0a-plugins-config-volume" pod "91f7307b-e9c5-45ee-871c-d1de04507c0a" (UID: "91f7307b-e9c5-45ee-871c-d1de04507c0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:50:28.997218 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:28.997196 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f7307b-e9c5-45ee-871c-d1de04507c0a-kube-api-access-t68jg" pod "91f7307b-e9c5-45ee-871c-d1de04507c0a" (UID: "91f7307b-e9c5-45ee-871c-d1de04507c0a"). InnerVolumeSpecName "kube-api-access-t68jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:50:29.096107 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:29.096070 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t68jg\" (UniqueName: \"kubernetes.io/projected/91f7307b-e9c5-45ee-871c-d1de04507c0a-kube-api-access-t68jg\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:29.096107 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:29.096102 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91f7307b-e9c5-45ee-871c-d1de04507c0a-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:29.540704 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:29.540665 4385 generic.go:356] "Generic (PLEG): container finished" podID="91f7307b-e9c5-45ee-871c-d1de04507c0a" containerID="d614f9ac349a5f1c1ead58a1f9ab330f50da70759400c9138f5ef3ce6aaaa40f" exitCode=0 Nov 01 06:50:29.540922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:29.540726 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j" Nov 01 06:50:29.540922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:29.540738 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j" event={"ID":"91f7307b-e9c5-45ee-871c-d1de04507c0a","Type":"ContainerDied","Data":"d614f9ac349a5f1c1ead58a1f9ab330f50da70759400c9138f5ef3ce6aaaa40f"} Nov 01 06:50:29.540922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:29.540796 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j" event={"ID":"91f7307b-e9c5-45ee-871c-d1de04507c0a","Type":"ContainerDied","Data":"250ab6cb1c1353469922a1d25ccd38ec33a01f09665b9c22afd73fcfd4aac4a9"} Nov 01 06:50:29.540922 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:29.540827 4385 scope.go:122] "RemoveContainer" containerID="d614f9ac349a5f1c1ead58a1f9ab330f50da70759400c9138f5ef3ce6aaaa40f" Nov 01 06:50:29.551990 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:29.551963 4385 scope.go:122] "RemoveContainer" containerID="d614f9ac349a5f1c1ead58a1f9ab330f50da70759400c9138f5ef3ce6aaaa40f" Nov 01 06:50:29.552329 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:50:29.552305 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d614f9ac349a5f1c1ead58a1f9ab330f50da70759400c9138f5ef3ce6aaaa40f\": container with ID starting with d614f9ac349a5f1c1ead58a1f9ab330f50da70759400c9138f5ef3ce6aaaa40f not found: ID does not exist" containerID="d614f9ac349a5f1c1ead58a1f9ab330f50da70759400c9138f5ef3ce6aaaa40f" Nov 01 06:50:29.552402 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:29.552343 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d614f9ac349a5f1c1ead58a1f9ab330f50da70759400c9138f5ef3ce6aaaa40f"} err="failed to get container status \"d614f9ac349a5f1c1ead58a1f9ab330f50da70759400c9138f5ef3ce6aaaa40f\": rpc error: code = NotFound desc = could not find container \"d614f9ac349a5f1c1ead58a1f9ab330f50da70759400c9138f5ef3ce6aaaa40f\": container with ID starting with d614f9ac349a5f1c1ead58a1f9ab330f50da70759400c9138f5ef3ce6aaaa40f not found: ID does not exist" Nov 01 06:50:29.677653 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:29.677619 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:29.708231 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:29.708193 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j"] Nov 01 06:50:29.711499 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:29.711478 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d855db-76f476689-zjv2j"] Nov 01 06:50:30.387167 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:30.387132 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91f7307b-e9c5-45ee-871c-d1de04507c0a" path="/var/lib/kubelet/pods/91f7307b-e9c5-45ee-871c-d1de04507c0a/volumes" Nov 01 06:50:30.936787 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:30.936744 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:33.017862 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:33.017802 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:33.210986 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:33.210955 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:35.626693 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:35.626661 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:36.782076 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:50:36.782030 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:50:40.260415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:40.260377 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:42.674499 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:42.674465 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:42.833637 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:42.833602 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:44.912682 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:44.912645 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:47.753855 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:47.753491 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:49.840647 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:49.840606 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:54.545448 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:54.545405 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:50:59.228295 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:50:59.228261 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:04.052732 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:04.052632 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:04.720124 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:04.720089 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:04.863156 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:04.863118 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:06.782043 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:06.782005 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:51:08.202447 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:08.202414 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:08.444185 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:08.444137 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:08.600340 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:08.600304 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:09.346452 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:09.346415 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:09.551698 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:09.551641 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:09.715025 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:09.714943 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:09.883699 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:09.883662 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:10.093724 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:10.093690 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:13.572689 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:13.572656 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:14.181716 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:14.181669 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:14.358800 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:14.358639 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:15.602582 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:15.602547 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:19.446895 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:19.446859 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:19.606626 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:19.606588 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:20.128686 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:20.128644 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:20.328369 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:20.328330 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:23.568883 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:23.568853 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp"] Nov 01 06:51:23.569313 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:23.569066 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" podUID="eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7" containerName="maas-api" containerID="cri-o://cc3e5848179574477e3a2efa31997d53ef938d3d5ff98c181ae7ad80023a6cf3" gracePeriod=30 Nov 01 06:51:23.686309 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:23.686238 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:23.885103 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:23.885077 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" Nov 01 06:51:24.029269 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:24.029238 4385 generic.go:356] "Generic (PLEG): container finished" podID="eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7" containerID="cc3e5848179574477e3a2efa31997d53ef938d3d5ff98c181ae7ad80023a6cf3" exitCode=0 Nov 01 06:51:24.029457 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:24.029285 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" Nov 01 06:51:24.029457 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:24.029323 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" event={"ID":"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7","Type":"ContainerDied","Data":"cc3e5848179574477e3a2efa31997d53ef938d3d5ff98c181ae7ad80023a6cf3"} Nov 01 06:51:24.029457 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:24.029394 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp" event={"ID":"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7","Type":"ContainerDied","Data":"1392fdf1903d0668db14c5cf457266044eb484b1d71e75cf643433dd25d09ef9"} Nov 01 06:51:24.029457 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:24.029429 4385 scope.go:122] "RemoveContainer" containerID="cc3e5848179574477e3a2efa31997d53ef938d3d5ff98c181ae7ad80023a6cf3" Nov 01 06:51:24.040759 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:24.040740 4385 scope.go:122] "RemoveContainer" containerID="cc3e5848179574477e3a2efa31997d53ef938d3d5ff98c181ae7ad80023a6cf3" Nov 01 06:51:24.041092 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:24.041064 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc3e5848179574477e3a2efa31997d53ef938d3d5ff98c181ae7ad80023a6cf3\": container with ID starting with cc3e5848179574477e3a2efa31997d53ef938d3d5ff98c181ae7ad80023a6cf3 not found: ID does not exist" containerID="cc3e5848179574477e3a2efa31997d53ef938d3d5ff98c181ae7ad80023a6cf3" Nov 01 06:51:24.041171 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:24.041107 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3e5848179574477e3a2efa31997d53ef938d3d5ff98c181ae7ad80023a6cf3"} err="failed to get container status \"cc3e5848179574477e3a2efa31997d53ef938d3d5ff98c181ae7ad80023a6cf3\": rpc error: code = NotFound desc = could not find container \"cc3e5848179574477e3a2efa31997d53ef938d3d5ff98c181ae7ad80023a6cf3\": container with ID starting with cc3e5848179574477e3a2efa31997d53ef938d3d5ff98c181ae7ad80023a6cf3 not found: ID does not exist" Nov 01 06:51:24.064686 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:24.064667 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-kube-api-access-8gvld\" (UniqueName: \"kubernetes.io/projected/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-kube-api-access-8gvld\") pod \"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7\" (UID: \"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7\") " Nov 01 06:51:24.064791 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:24.064701 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-maas-api-tls\") pod \"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7\" (UID: \"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7\") " Nov 01 06:51:24.064791 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:24.064737 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-openshift-service-ca\") pod \"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7\" (UID: \"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7\") " Nov 01 06:51:24.064895 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:24.064792 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-metrics-certs\") pod \"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7\" (UID: \"eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7\") " Nov 01 06:51:24.065206 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:24.065178 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-openshift-service-ca" pod "eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7" (UID: "eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:51:24.066994 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:24.066965 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-metrics-certs" pod "eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7" (UID: "eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:51:24.067082 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:24.066995 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-maas-api-tls" pod "eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7" (UID: "eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:51:24.067082 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:24.067070 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-kube-api-access-8gvld" pod "eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7" (UID: "eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7"). InnerVolumeSpecName "kube-api-access-8gvld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:51:24.165981 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:24.165907 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gvld\" (UniqueName: \"kubernetes.io/projected/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-kube-api-access-8gvld\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:24.165981 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:24.165937 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-maas-api-tls\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:24.165981 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:24.165947 4385 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-openshift-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:24.165981 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:24.165956 4385 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7-metrics-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:24.450606 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:24.450521 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp"] Nov 01 06:51:24.453563 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:24.453538 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d855db-774554c59-jkjfp"] Nov 01 06:51:25.428353 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:25.428319 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:26.222873 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:26.220430 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:26.406857 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:26.406506 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7" path="/var/lib/kubelet/pods/eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7/volumes" Nov 01 06:51:26.418226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:26.418156 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:26.980334 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:26.980298 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-985777bbf-wxk2d"] Nov 01 06:51:26.980814 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:26.980801 4385 cpu_manager.go:414] "containerMap: removing container" podUID="eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7" containerName="maas-api" Nov 01 06:51:26.980900 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:26.980815 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7" containerName="maas-api" Nov 01 06:51:26.980900 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:26.980843 4385 cpu_manager.go:414] "containerMap: removing container" podUID="91f7307b-e9c5-45ee-871c-d1de04507c0a" containerName="payload-processing" Nov 01 06:51:26.980900 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:26.980848 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91f7307b-e9c5-45ee-871c-d1de04507c0a" containerName="payload-processing" Nov 01 06:51:26.980999 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:26.980917 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaaaefbd-b8c6-4179-8ce1-f492ccfde1f7" containerName="maas-api" Nov 01 06:51:26.980999 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:26.980927 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="91f7307b-e9c5-45ee-871c-d1de04507c0a" containerName="payload-processing" Nov 01 06:51:26.985703 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:26.985688 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-985777bbf-wxk2d" Nov 01 06:51:26.993604 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:26.993584 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-routing-overlay-volume\") pod \"payload-processing-external-model-985777bbf-wxk2d\" (UID: \"1a9ead09-b6ad-4ce4-98ed-1c337420b59c\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-wxk2d" Nov 01 06:51:26.993691 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:26.993633 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-plugins-config-volume\") pod \"payload-processing-external-model-985777bbf-wxk2d\" (UID: \"1a9ead09-b6ad-4ce4-98ed-1c337420b59c\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-wxk2d" Nov 01 06:51:26.993726 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:26.993715 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-provider-credentials\") pod \"payload-processing-external-model-985777bbf-wxk2d\" (UID: \"1a9ead09-b6ad-4ce4-98ed-1c337420b59c\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-wxk2d" Nov 01 06:51:26.993792 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:26.993780 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn9lw\" (UniqueName: \"kubernetes.io/projected/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-kube-api-access-bn9lw\") pod \"payload-processing-external-model-985777bbf-wxk2d\" (UID: \"1a9ead09-b6ad-4ce4-98ed-1c337420b59c\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-wxk2d" Nov 01 06:51:27.088341 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:27.088306 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1b7c4c0142f1\"" Nov 01 06:51:27.088341 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:27.088336 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 01 06:51:27.088572 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:27.088311 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 01 06:51:27.088572 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:27.088311 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-75fbcfa535aa\"" Nov 01 06:51:27.088572 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:27.088388 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 01 06:51:27.088572 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:27.088393 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-bad5b4f3f7d2\"" Nov 01 06:51:27.088572 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:27.088398 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ba171a9860eb\"" Nov 01 06:51:27.088572 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:27.088415 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-tng78\"" Nov 01 06:51:27.088572 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:27.088453 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 01 06:51:27.091870 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:27.091850 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-985777bbf-wxk2d"] Nov 01 06:51:27.094431 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:27.094407 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-provider-credentials\") pod \"payload-processing-external-model-985777bbf-wxk2d\" (UID: \"1a9ead09-b6ad-4ce4-98ed-1c337420b59c\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-wxk2d" Nov 01 06:51:27.094562 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:27.094492 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn9lw\" (UniqueName: \"kubernetes.io/projected/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-kube-api-access-bn9lw\") pod \"payload-processing-external-model-985777bbf-wxk2d\" (UID: \"1a9ead09-b6ad-4ce4-98ed-1c337420b59c\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-wxk2d" Nov 01 06:51:27.094562 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:27.094550 4385 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-75fbcfa535aa: secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:27.094672 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:27.094568 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-985777bbf-wxk2d: secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:27.094672 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:27.094577 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-routing-overlay-volume\") pod \"payload-processing-external-model-985777bbf-wxk2d\" (UID: \"1a9ead09-b6ad-4ce4-98ed-1c337420b59c\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-wxk2d" Nov 01 06:51:27.094672 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:27.094620 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-provider-credentials podName:1a9ead09-b6ad-4ce4-98ed-1c337420b59c nodeName:}" failed. No retries permitted until 2026-11-01 06:51:27.594605288 +0000 UTC m=+2497.358405123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-provider-credentials") pod "payload-processing-external-model-985777bbf-wxk2d" (UID: "1a9ead09-b6ad-4ce4-98ed-1c337420b59c") : secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:27.094861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:27.094678 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-plugins-config-volume\") pod \"payload-processing-external-model-985777bbf-wxk2d\" (UID: \"1a9ead09-b6ad-4ce4-98ed-1c337420b59c\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-wxk2d" Nov 01 06:51:27.095274 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:27.095250 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-routing-overlay-volume\") pod \"payload-processing-external-model-985777bbf-wxk2d\" (UID: \"1a9ead09-b6ad-4ce4-98ed-1c337420b59c\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-wxk2d" Nov 01 06:51:27.095359 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:27.095336 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-plugins-config-volume\") pod \"payload-processing-external-model-985777bbf-wxk2d\" (UID: \"1a9ead09-b6ad-4ce4-98ed-1c337420b59c\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-wxk2d" Nov 01 06:51:27.155663 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:27.155630 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn9lw\" (UniqueName: \"kubernetes.io/projected/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-kube-api-access-bn9lw\") pod \"payload-processing-external-model-985777bbf-wxk2d\" (UID: \"1a9ead09-b6ad-4ce4-98ed-1c337420b59c\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-wxk2d" Nov 01 06:51:27.598968 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:27.598932 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-provider-credentials\") pod \"payload-processing-external-model-985777bbf-wxk2d\" (UID: \"1a9ead09-b6ad-4ce4-98ed-1c337420b59c\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-wxk2d" Nov 01 06:51:27.599189 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:27.599094 4385 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-75fbcfa535aa: secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:27.599189 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:27.599124 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-985777bbf-wxk2d: secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:27.599320 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:27.599196 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-provider-credentials podName:1a9ead09-b6ad-4ce4-98ed-1c337420b59c nodeName:}" failed. No retries permitted until 2026-11-01 06:51:28.599177521 +0000 UTC m=+2498.362977372 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-provider-credentials") pod "payload-processing-external-model-985777bbf-wxk2d" (UID: "1a9ead09-b6ad-4ce4-98ed-1c337420b59c") : secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:28.062120 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:28.062021 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:28.222419 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:28.222375 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:28.606969 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:28.606942 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-provider-credentials\") pod \"payload-processing-external-model-985777bbf-wxk2d\" (UID: \"1a9ead09-b6ad-4ce4-98ed-1c337420b59c\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-wxk2d" Nov 01 06:51:28.607140 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:28.607101 4385 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-75fbcfa535aa: secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:28.607140 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:28.607124 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-985777bbf-wxk2d: secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:28.607249 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:28.607175 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-provider-credentials podName:1a9ead09-b6ad-4ce4-98ed-1c337420b59c nodeName:}" failed. No retries permitted until 2026-11-01 06:51:30.607161116 +0000 UTC m=+2500.370960951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-provider-credentials") pod "payload-processing-external-model-985777bbf-wxk2d" (UID: "1a9ead09-b6ad-4ce4-98ed-1c337420b59c") : secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:29.383299 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:29.383266 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-985777bbf-wxk2d"] Nov 01 06:51:29.383689 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:29.383557 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-985777bbf-wxk2d" podUID="1a9ead09-b6ad-4ce4-98ed-1c337420b59c" Nov 01 06:51:29.414034 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:29.413994 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f"] Nov 01 06:51:29.418757 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:29.418733 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" Nov 01 06:51:29.496262 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:29.496229 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f"] Nov 01 06:51:29.514961 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:29.514931 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvwgx\" (UniqueName: \"kubernetes.io/projected/f5dec640-0c5c-483a-b87f-6aba894a88cc-kube-api-access-jvwgx\") pod \"payload-processing-external-model-866d5b9c86-lp22f\" (UID: \"f5dec640-0c5c-483a-b87f-6aba894a88cc\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" Nov 01 06:51:29.515122 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:29.515065 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5dec640-0c5c-483a-b87f-6aba894a88cc-provider-credentials\") pod \"payload-processing-external-model-866d5b9c86-lp22f\" (UID: \"f5dec640-0c5c-483a-b87f-6aba894a88cc\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" Nov 01 06:51:29.515122 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:29.515113 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5dec640-0c5c-483a-b87f-6aba894a88cc-routing-overlay-volume\") pod \"payload-processing-external-model-866d5b9c86-lp22f\" (UID: \"f5dec640-0c5c-483a-b87f-6aba894a88cc\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" Nov 01 06:51:29.515254 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:29.515141 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5dec640-0c5c-483a-b87f-6aba894a88cc-plugins-config-volume\") pod \"payload-processing-external-model-866d5b9c86-lp22f\" (UID: \"f5dec640-0c5c-483a-b87f-6aba894a88cc\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" Nov 01 06:51:29.616824 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:29.616793 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5dec640-0c5c-483a-b87f-6aba894a88cc-provider-credentials\") pod \"payload-processing-external-model-866d5b9c86-lp22f\" (UID: \"f5dec640-0c5c-483a-b87f-6aba894a88cc\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" Nov 01 06:51:29.617019 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:29.616875 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5dec640-0c5c-483a-b87f-6aba894a88cc-routing-overlay-volume\") pod \"payload-processing-external-model-866d5b9c86-lp22f\" (UID: \"f5dec640-0c5c-483a-b87f-6aba894a88cc\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" Nov 01 06:51:29.617019 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:29.616922 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5dec640-0c5c-483a-b87f-6aba894a88cc-plugins-config-volume\") pod \"payload-processing-external-model-866d5b9c86-lp22f\" (UID: \"f5dec640-0c5c-483a-b87f-6aba894a88cc\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" Nov 01 06:51:29.617019 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:29.617004 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvwgx\" (UniqueName: \"kubernetes.io/projected/f5dec640-0c5c-483a-b87f-6aba894a88cc-kube-api-access-jvwgx\") pod \"payload-processing-external-model-866d5b9c86-lp22f\" (UID: \"f5dec640-0c5c-483a-b87f-6aba894a88cc\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" Nov 01 06:51:29.617675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:29.617600 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5dec640-0c5c-483a-b87f-6aba894a88cc-routing-overlay-volume\") pod \"payload-processing-external-model-866d5b9c86-lp22f\" (UID: \"f5dec640-0c5c-483a-b87f-6aba894a88cc\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" Nov 01 06:51:29.617675 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:29.617659 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5dec640-0c5c-483a-b87f-6aba894a88cc-plugins-config-volume\") pod \"payload-processing-external-model-866d5b9c86-lp22f\" (UID: \"f5dec640-0c5c-483a-b87f-6aba894a88cc\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" Nov 01 06:51:29.619745 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:29.619721 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5dec640-0c5c-483a-b87f-6aba894a88cc-provider-credentials\") pod \"payload-processing-external-model-866d5b9c86-lp22f\" (UID: \"f5dec640-0c5c-483a-b87f-6aba894a88cc\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" Nov 01 06:51:29.659531 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:29.659437 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:29.673473 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:29.673445 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvwgx\" (UniqueName: \"kubernetes.io/projected/f5dec640-0c5c-483a-b87f-6aba894a88cc-kube-api-access-jvwgx\") pod \"payload-processing-external-model-866d5b9c86-lp22f\" (UID: \"f5dec640-0c5c-483a-b87f-6aba894a88cc\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" Nov 01 06:51:29.732247 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:29.732207 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" Nov 01 06:51:29.867855 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:29.866454 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:30.035208 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:30.035105 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:30.084888 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:30.084860 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-985777bbf-wxk2d" Nov 01 06:51:30.114874 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:30.101955 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-985777bbf-wxk2d" Nov 01 06:51:30.123541 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:30.123514 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-plugins-config-volume\") pod \"1a9ead09-b6ad-4ce4-98ed-1c337420b59c\" (UID: \"1a9ead09-b6ad-4ce4-98ed-1c337420b59c\") " Nov 01 06:51:30.123900 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:30.123875 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-kube-api-access-bn9lw\" (UniqueName: \"kubernetes.io/projected/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-kube-api-access-bn9lw\") pod \"1a9ead09-b6ad-4ce4-98ed-1c337420b59c\" (UID: \"1a9ead09-b6ad-4ce4-98ed-1c337420b59c\") " Nov 01 06:51:30.124114 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:30.124097 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-routing-overlay-volume\") pod \"1a9ead09-b6ad-4ce4-98ed-1c337420b59c\" (UID: \"1a9ead09-b6ad-4ce4-98ed-1c337420b59c\") " Nov 01 06:51:30.125854 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:30.125812 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-routing-overlay-volume" pod "1a9ead09-b6ad-4ce4-98ed-1c337420b59c" (UID: "1a9ead09-b6ad-4ce4-98ed-1c337420b59c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:51:30.127146 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:30.127123 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-plugins-config-volume" pod "1a9ead09-b6ad-4ce4-98ed-1c337420b59c" (UID: "1a9ead09-b6ad-4ce4-98ed-1c337420b59c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:51:30.128530 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:30.128443 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f"] Nov 01 06:51:30.134911 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:30.133418 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-kube-api-access-bn9lw" pod "1a9ead09-b6ad-4ce4-98ed-1c337420b59c" (UID: "1a9ead09-b6ad-4ce4-98ed-1c337420b59c"). InnerVolumeSpecName "kube-api-access-bn9lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:51:30.184556 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:30.184522 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:30.226699 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:30.226672 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:30.226699 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:30.226700 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn9lw\" (UniqueName: \"kubernetes.io/projected/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-kube-api-access-bn9lw\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:30.226864 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:30.226709 4385 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-routing-overlay-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:30.495776 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:30.495742 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:30.631812 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:30.631779 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-provider-credentials\") pod \"payload-processing-external-model-985777bbf-wxk2d\" (UID: \"1a9ead09-b6ad-4ce4-98ed-1c337420b59c\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-wxk2d" Nov 01 06:51:30.631988 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:30.631948 4385 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-75fbcfa535aa: secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:30.631988 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:30.631979 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-985777bbf-wxk2d: secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:30.632074 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:30.632033 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-provider-credentials podName:1a9ead09-b6ad-4ce4-98ed-1c337420b59c nodeName:}" failed. No retries permitted until 2026-11-01 06:51:34.632019748 +0000 UTC m=+2504.395819590 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-provider-credentials") pod "payload-processing-external-model-985777bbf-wxk2d" (UID: "1a9ead09-b6ad-4ce4-98ed-1c337420b59c") : secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:30.758014 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:30.757940 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:30.904615 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:30.904548 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:31.092713 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:31.092685 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-985777bbf-wxk2d" Nov 01 06:51:31.092959 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:31.092713 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" event={"ID":"f5dec640-0c5c-483a-b87f-6aba894a88cc","Type":"ContainerStarted","Data":"054721809f8df7f7d6677f5421c63df1922f32ebf3a514d6db47ac78343d9d25"} Nov 01 06:51:31.092959 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:31.092771 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" event={"ID":"f5dec640-0c5c-483a-b87f-6aba894a88cc","Type":"ContainerStarted","Data":"765d96032bb74eaa8ce7a642ea7296c5f4615b85914bae86000e8d179ac70bb1"} Nov 01 06:51:31.093098 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:31.093021 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" Nov 01 06:51:31.206764 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:31.206710 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" podStartSLOduration=2.206696641 podStartE2EDuration="2.206696641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:51:31.204945742 +0000 UTC m=+2500.968745611" watchObservedRunningTime="2026-11-01 06:51:31.206696641 +0000 UTC m=+2500.970496526" Nov 01 06:51:31.296072 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:31.296040 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:31.377222 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:31.377134 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-985777bbf-wxk2d"] Nov 01 06:51:31.381416 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:31.381385 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-985777bbf-wxk2d"] Nov 01 06:51:31.440354 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:31.440292 4385 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a9ead09-b6ad-4ce4-98ed-1c337420b59c-provider-credentials\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:31.900514 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:31.900326 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:32.154659 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:32.154535 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:32.384435 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:32.384405 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a9ead09-b6ad-4ce4-98ed-1c337420b59c" path="/var/lib/kubelet/pods/1a9ead09-b6ad-4ce4-98ed-1c337420b59c/volumes" Nov 01 06:51:34.240120 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:34.240076 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f"] Nov 01 06:51:34.262465 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:34.262428 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4"] Nov 01 06:51:34.266518 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:34.266501 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" Nov 01 06:51:34.319071 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:34.319044 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0121f247d8fb\"" Nov 01 06:51:34.323818 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:34.323789 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4"] Nov 01 06:51:34.379780 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:34.379747 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f499a0-aa4a-4a99-af16-2d3775701b21-plugins-config-volume\") pod \"payload-processing-external-model-6bd55dd7d7-7swd4\" (UID: \"73f499a0-aa4a-4a99-af16-2d3775701b21\") " pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" Nov 01 06:51:34.380137 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:34.380109 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-provider-credentials\") pod \"payload-processing-external-model-6bd55dd7d7-7swd4\" (UID: \"73f499a0-aa4a-4a99-af16-2d3775701b21\") " pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" Nov 01 06:51:34.380257 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:34.380192 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x85sl\" (UniqueName: \"kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-kube-api-access-x85sl\") pod \"payload-processing-external-model-6bd55dd7d7-7swd4\" (UID: \"73f499a0-aa4a-4a99-af16-2d3775701b21\") " pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" Nov 01 06:51:34.380408 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:34.380366 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/73f499a0-aa4a-4a99-af16-2d3775701b21-routing-overlay-volume\") pod \"payload-processing-external-model-6bd55dd7d7-7swd4\" (UID: \"73f499a0-aa4a-4a99-af16-2d3775701b21\") " pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" Nov 01 06:51:34.481912 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:34.481877 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-provider-credentials\") pod \"payload-processing-external-model-6bd55dd7d7-7swd4\" (UID: \"73f499a0-aa4a-4a99-af16-2d3775701b21\") " pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" Nov 01 06:51:34.482125 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:34.481986 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x85sl\" (UniqueName: \"kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-kube-api-access-x85sl\") pod \"payload-processing-external-model-6bd55dd7d7-7swd4\" (UID: \"73f499a0-aa4a-4a99-af16-2d3775701b21\") " pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" Nov 01 06:51:34.482125 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:34.482103 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/73f499a0-aa4a-4a99-af16-2d3775701b21-routing-overlay-volume\") pod \"payload-processing-external-model-6bd55dd7d7-7swd4\" (UID: \"73f499a0-aa4a-4a99-af16-2d3775701b21\") " pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" Nov 01 06:51:34.482516 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:34.482358 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f499a0-aa4a-4a99-af16-2d3775701b21-plugins-config-volume\") pod \"payload-processing-external-model-6bd55dd7d7-7swd4\" (UID: \"73f499a0-aa4a-4a99-af16-2d3775701b21\") " pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" Nov 01 06:51:34.483074 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:34.483051 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/73f499a0-aa4a-4a99-af16-2d3775701b21-routing-overlay-volume\") pod \"payload-processing-external-model-6bd55dd7d7-7swd4\" (UID: \"73f499a0-aa4a-4a99-af16-2d3775701b21\") " pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" Nov 01 06:51:34.483139 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:34.483058 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f499a0-aa4a-4a99-af16-2d3775701b21-plugins-config-volume\") pod \"payload-processing-external-model-6bd55dd7d7-7swd4\" (UID: \"73f499a0-aa4a-4a99-af16-2d3775701b21\") " pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" Nov 01 06:51:34.485441 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:34.485419 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-provider-credentials\") pod \"payload-processing-external-model-6bd55dd7d7-7swd4\" (UID: \"73f499a0-aa4a-4a99-af16-2d3775701b21\") " pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" Nov 01 06:51:34.538008 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:34.537951 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x85sl\" (UniqueName: \"kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-kube-api-access-x85sl\") pod \"payload-processing-external-model-6bd55dd7d7-7swd4\" (UID: \"73f499a0-aa4a-4a99-af16-2d3775701b21\") " pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" Nov 01 06:51:34.580497 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:34.580464 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" Nov 01 06:51:34.956180 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:34.956151 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4"] Nov 01 06:51:34.956319 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:51:34.956294 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f499a0_aa4a_4a99_af16_2d3775701b21.slice/crio-ecee5ba2b0639df1e7eb35861072ffbd4af6c43d32f15a23f246cd0c3f0825a0 WatchSource:0}: Error finding container ecee5ba2b0639df1e7eb35861072ffbd4af6c43d32f15a23f246cd0c3f0825a0: Status 404 returned error can't find the container with id ecee5ba2b0639df1e7eb35861072ffbd4af6c43d32f15a23f246cd0c3f0825a0 Nov 01 06:51:35.118309 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:35.118272 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" event={"ID":"73f499a0-aa4a-4a99-af16-2d3775701b21","Type":"ContainerStarted","Data":"a954d50d6221a8b19b0da6d3eda78586054cc3aa1bf2c52beabb5012d6c0ee0c"} Nov 01 06:51:35.118482 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:35.118340 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" Nov 01 06:51:35.118482 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:35.118360 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" event={"ID":"73f499a0-aa4a-4a99-af16-2d3775701b21","Type":"ContainerStarted","Data":"ecee5ba2b0639df1e7eb35861072ffbd4af6c43d32f15a23f246cd0c3f0825a0"} Nov 01 06:51:35.234890 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:35.234793 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" podStartSLOduration=1.234780478 podStartE2EDuration="1.234780478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:51:35.231298914 +0000 UTC m=+2504.995098783" watchObservedRunningTime="2026-11-01 06:51:35.234780478 +0000 UTC m=+2504.998580360" Nov 01 06:51:35.537514 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:35.537428 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:35.685724 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:35.685685 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:36.100140 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:36.100108 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:36.203572 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:36.203539 4385 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0121f247d8fb: secret "agc-cred-0121f247d8fb" not found Nov 01 06:51:36.203572 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:36.203576 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4: secret "agc-cred-0121f247d8fb" not found Nov 01 06:51:36.203853 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:36.203633 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-provider-credentials podName:73f499a0-aa4a-4a99-af16-2d3775701b21 nodeName:}" failed. No retries permitted until 2026-11-01 06:51:36.70361257 +0000 UTC m=+2506.467412407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-provider-credentials") pod "payload-processing-external-model-6bd55dd7d7-7swd4" (UID: "73f499a0-aa4a-4a99-af16-2d3775701b21") : secret "agc-cred-0121f247d8fb" not found Nov 01 06:51:36.290638 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:36.290605 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:36.463723 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:36.463621 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:36.625336 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:36.625292 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:36.708924 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:36.708894 4385 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0121f247d8fb: secret "agc-cred-0121f247d8fb" not found Nov 01 06:51:36.708924 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:36.708928 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4: secret "agc-cred-0121f247d8fb" not found Nov 01 06:51:36.709125 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:36.708983 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-provider-credentials podName:73f499a0-aa4a-4a99-af16-2d3775701b21 nodeName:}" failed. No retries permitted until 2026-11-01 06:51:37.708968726 +0000 UTC m=+2507.472768585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-provider-credentials") pod "payload-processing-external-model-6bd55dd7d7-7swd4" (UID: "73f499a0-aa4a-4a99-af16-2d3775701b21") : secret "agc-cred-0121f247d8fb" not found Nov 01 06:51:36.780326 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:36.780243 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:51:37.720563 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:37.720528 4385 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0121f247d8fb: secret "agc-cred-0121f247d8fb" not found Nov 01 06:51:37.720563 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:37.720560 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4: secret "agc-cred-0121f247d8fb" not found Nov 01 06:51:37.721022 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:37.720615 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-provider-credentials podName:73f499a0-aa4a-4a99-af16-2d3775701b21 nodeName:}" failed. No retries permitted until 2026-11-01 06:51:39.720601275 +0000 UTC m=+2509.484401111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-provider-credentials") pod "payload-processing-external-model-6bd55dd7d7-7swd4" (UID: "73f499a0-aa4a-4a99-af16-2d3775701b21") : secret "agc-cred-0121f247d8fb" not found Nov 01 06:51:38.849827 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:38.849792 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:39.022856 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.022796 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4"] Nov 01 06:51:39.073032 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.073006 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q"] Nov 01 06:51:39.077056 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.077036 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" Nov 01 06:51:39.136960 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.136867 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q"] Nov 01 06:51:39.139104 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.139077 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b0f033-c464-4bab-b201-07686045040e-plugins-config-volume\") pod \"payload-processing-external-model-866d5b9c86-dxh9q\" (UID: \"71b0f033-c464-4bab-b201-07686045040e\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" Nov 01 06:51:39.139269 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.139213 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5p9n\" (UniqueName: \"kubernetes.io/projected/71b0f033-c464-4bab-b201-07686045040e-kube-api-access-n5p9n\") pod \"payload-processing-external-model-866d5b9c86-dxh9q\" (UID: \"71b0f033-c464-4bab-b201-07686045040e\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" Nov 01 06:51:39.139376 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.139358 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71b0f033-c464-4bab-b201-07686045040e-routing-overlay-volume\") pod \"payload-processing-external-model-866d5b9c86-dxh9q\" (UID: \"71b0f033-c464-4bab-b201-07686045040e\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" Nov 01 06:51:39.139496 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.139423 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71b0f033-c464-4bab-b201-07686045040e-provider-credentials\") pod \"payload-processing-external-model-866d5b9c86-dxh9q\" (UID: \"71b0f033-c464-4bab-b201-07686045040e\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" Nov 01 06:51:39.240630 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.240588 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" podUID="f5dec640-0c5c-483a-b87f-6aba894a88cc" containerName="payload-processing" containerID="cri-o://054721809f8df7f7d6677f5421c63df1922f32ebf3a514d6db47ac78343d9d25" gracePeriod=25 Nov 01 06:51:39.241403 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.241141 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71b0f033-c464-4bab-b201-07686045040e-routing-overlay-volume\") pod \"payload-processing-external-model-866d5b9c86-dxh9q\" (UID: \"71b0f033-c464-4bab-b201-07686045040e\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" Nov 01 06:51:39.241403 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.241192 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71b0f033-c464-4bab-b201-07686045040e-provider-credentials\") pod \"payload-processing-external-model-866d5b9c86-dxh9q\" (UID: \"71b0f033-c464-4bab-b201-07686045040e\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" Nov 01 06:51:39.241403 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.241262 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b0f033-c464-4bab-b201-07686045040e-plugins-config-volume\") pod \"payload-processing-external-model-866d5b9c86-dxh9q\" (UID: \"71b0f033-c464-4bab-b201-07686045040e\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" Nov 01 06:51:39.241403 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.241338 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5p9n\" (UniqueName: \"kubernetes.io/projected/71b0f033-c464-4bab-b201-07686045040e-kube-api-access-n5p9n\") pod \"payload-processing-external-model-866d5b9c86-dxh9q\" (UID: \"71b0f033-c464-4bab-b201-07686045040e\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" Nov 01 06:51:39.241959 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.241933 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71b0f033-c464-4bab-b201-07686045040e-routing-overlay-volume\") pod \"payload-processing-external-model-866d5b9c86-dxh9q\" (UID: \"71b0f033-c464-4bab-b201-07686045040e\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" Nov 01 06:51:39.242501 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.242478 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b0f033-c464-4bab-b201-07686045040e-plugins-config-volume\") pod \"payload-processing-external-model-866d5b9c86-dxh9q\" (UID: \"71b0f033-c464-4bab-b201-07686045040e\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" Nov 01 06:51:39.244350 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.244323 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71b0f033-c464-4bab-b201-07686045040e-provider-credentials\") pod \"payload-processing-external-model-866d5b9c86-dxh9q\" (UID: \"71b0f033-c464-4bab-b201-07686045040e\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" Nov 01 06:51:39.298464 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.298414 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5p9n\" (UniqueName: \"kubernetes.io/projected/71b0f033-c464-4bab-b201-07686045040e-kube-api-access-n5p9n\") pod \"payload-processing-external-model-866d5b9c86-dxh9q\" (UID: \"71b0f033-c464-4bab-b201-07686045040e\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" Nov 01 06:51:39.446916 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.428863 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" Nov 01 06:51:39.460502 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.454003 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:39.712515 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.712488 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" Nov 01 06:51:39.760047 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.760002 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5dec640-0c5c-483a-b87f-6aba894a88cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5dec640-0c5c-483a-b87f-6aba894a88cc-plugins-config-volume\") pod \"f5dec640-0c5c-483a-b87f-6aba894a88cc\" (UID: \"f5dec640-0c5c-483a-b87f-6aba894a88cc\") " Nov 01 06:51:39.760264 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.760059 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5dec640-0c5c-483a-b87f-6aba894a88cc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5dec640-0c5c-483a-b87f-6aba894a88cc-routing-overlay-volume\") pod \"f5dec640-0c5c-483a-b87f-6aba894a88cc\" (UID: \"f5dec640-0c5c-483a-b87f-6aba894a88cc\") " Nov 01 06:51:39.760264 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.760163 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5dec640-0c5c-483a-b87f-6aba894a88cc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5dec640-0c5c-483a-b87f-6aba894a88cc-provider-credentials\") pod \"f5dec640-0c5c-483a-b87f-6aba894a88cc\" (UID: \"f5dec640-0c5c-483a-b87f-6aba894a88cc\") " Nov 01 06:51:39.760388 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.760266 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5dec640-0c5c-483a-b87f-6aba894a88cc-kube-api-access-jvwgx\" (UniqueName: \"kubernetes.io/projected/f5dec640-0c5c-483a-b87f-6aba894a88cc-kube-api-access-jvwgx\") pod \"f5dec640-0c5c-483a-b87f-6aba894a88cc\" (UID: \"f5dec640-0c5c-483a-b87f-6aba894a88cc\") " Nov 01 06:51:39.760819 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:39.760797 4385 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0121f247d8fb: secret "agc-cred-0121f247d8fb" not found Nov 01 06:51:39.760943 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:39.760854 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4: secret "agc-cred-0121f247d8fb" not found Nov 01 06:51:39.760943 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:39.760915 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-provider-credentials podName:73f499a0-aa4a-4a99-af16-2d3775701b21 nodeName:}" failed. No retries permitted until 2026-11-01 06:51:43.760895845 +0000 UTC m=+2513.524695700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-provider-credentials") pod "payload-processing-external-model-6bd55dd7d7-7swd4" (UID: "73f499a0-aa4a-4a99-af16-2d3775701b21") : secret "agc-cred-0121f247d8fb" not found Nov 01 06:51:39.761938 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.761911 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5dec640-0c5c-483a-b87f-6aba894a88cc-routing-overlay-volume" pod "f5dec640-0c5c-483a-b87f-6aba894a88cc" (UID: "f5dec640-0c5c-483a-b87f-6aba894a88cc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:51:39.764496 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.762220 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5dec640-0c5c-483a-b87f-6aba894a88cc-plugins-config-volume" pod "f5dec640-0c5c-483a-b87f-6aba894a88cc" (UID: "f5dec640-0c5c-483a-b87f-6aba894a88cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:51:39.773084 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.773043 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5dec640-0c5c-483a-b87f-6aba894a88cc-kube-api-access-jvwgx" pod "f5dec640-0c5c-483a-b87f-6aba894a88cc" (UID: "f5dec640-0c5c-483a-b87f-6aba894a88cc"). InnerVolumeSpecName "kube-api-access-jvwgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:51:39.775034 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.775001 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5dec640-0c5c-483a-b87f-6aba894a88cc-provider-credentials" pod "f5dec640-0c5c-483a-b87f-6aba894a88cc" (UID: "f5dec640-0c5c-483a-b87f-6aba894a88cc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:51:39.841586 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.841544 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:39.862367 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.862330 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvwgx\" (UniqueName: \"kubernetes.io/projected/f5dec640-0c5c-483a-b87f-6aba894a88cc-kube-api-access-jvwgx\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:39.862367 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.862371 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5dec640-0c5c-483a-b87f-6aba894a88cc-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:39.862793 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.862386 4385 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5dec640-0c5c-483a-b87f-6aba894a88cc-routing-overlay-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:39.862793 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.862402 4385 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5dec640-0c5c-483a-b87f-6aba894a88cc-provider-credentials\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:39.953521 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:51:39.953494 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b0f033_c464_4bab_b201_07686045040e.slice/crio-16fe3ccf4bed3842ad80c5893565addb649a406bee6d6712d304f3b84081e10f WatchSource:0}: Error finding container 16fe3ccf4bed3842ad80c5893565addb649a406bee6d6712d304f3b84081e10f: Status 404 returned error can't find the container with id 16fe3ccf4bed3842ad80c5893565addb649a406bee6d6712d304f3b84081e10f Nov 01 06:51:39.953813 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:39.953792 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q"] Nov 01 06:51:40.159327 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:40.159289 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" event={"ID":"71b0f033-c464-4bab-b201-07686045040e","Type":"ContainerStarted","Data":"91b985cf70b13f002b4ab8f7f2fe1839a5de57e0d6541366b3322e67e117f3b2"} Nov 01 06:51:40.159514 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:40.159352 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" event={"ID":"71b0f033-c464-4bab-b201-07686045040e","Type":"ContainerStarted","Data":"16fe3ccf4bed3842ad80c5893565addb649a406bee6d6712d304f3b84081e10f"} Nov 01 06:51:40.159514 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:40.159491 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" Nov 01 06:51:40.160772 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:40.160744 4385 generic.go:356] "Generic (PLEG): container finished" podID="f5dec640-0c5c-483a-b87f-6aba894a88cc" containerID="054721809f8df7f7d6677f5421c63df1922f32ebf3a514d6db47ac78343d9d25" exitCode=0 Nov 01 06:51:40.160933 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:40.160784 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" Nov 01 06:51:40.160933 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:40.160810 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" event={"ID":"f5dec640-0c5c-483a-b87f-6aba894a88cc","Type":"ContainerDied","Data":"054721809f8df7f7d6677f5421c63df1922f32ebf3a514d6db47ac78343d9d25"} Nov 01 06:51:40.160933 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:40.160872 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f" event={"ID":"f5dec640-0c5c-483a-b87f-6aba894a88cc","Type":"ContainerDied","Data":"765d96032bb74eaa8ce7a642ea7296c5f4615b85914bae86000e8d179ac70bb1"} Nov 01 06:51:40.160933 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:40.160893 4385 scope.go:122] "RemoveContainer" containerID="054721809f8df7f7d6677f5421c63df1922f32ebf3a514d6db47ac78343d9d25" Nov 01 06:51:40.172484 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:40.172464 4385 scope.go:122] "RemoveContainer" containerID="054721809f8df7f7d6677f5421c63df1922f32ebf3a514d6db47ac78343d9d25" Nov 01 06:51:40.172786 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:40.172766 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"054721809f8df7f7d6677f5421c63df1922f32ebf3a514d6db47ac78343d9d25\": container with ID starting with 054721809f8df7f7d6677f5421c63df1922f32ebf3a514d6db47ac78343d9d25 not found: ID does not exist" containerID="054721809f8df7f7d6677f5421c63df1922f32ebf3a514d6db47ac78343d9d25" Nov 01 06:51:40.172896 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:40.172794 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"054721809f8df7f7d6677f5421c63df1922f32ebf3a514d6db47ac78343d9d25"} err="failed to get container status \"054721809f8df7f7d6677f5421c63df1922f32ebf3a514d6db47ac78343d9d25\": rpc error: code = NotFound desc = could not find container \"054721809f8df7f7d6677f5421c63df1922f32ebf3a514d6db47ac78343d9d25\": container with ID starting with 054721809f8df7f7d6677f5421c63df1922f32ebf3a514d6db47ac78343d9d25 not found: ID does not exist" Nov 01 06:51:40.277996 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:40.277934 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podStartSLOduration=1.277919214 podStartE2EDuration="1.277919214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:51:40.272657619 +0000 UTC m=+2510.036457478" watchObservedRunningTime="2026-11-01 06:51:40.277919214 +0000 UTC m=+2510.041719070" Nov 01 06:51:40.389935 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:40.389901 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f"] Nov 01 06:51:40.394344 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:40.394322 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-lp22f"] Nov 01 06:51:41.319483 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:41.319444 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:41.796345 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:41.796308 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:41.965627 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:41.965587 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:42.195094 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.195060 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb"] Nov 01 06:51:42.195604 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.195592 4385 cpu_manager.go:414] "containerMap: removing container" podUID="f5dec640-0c5c-483a-b87f-6aba894a88cc" containerName="payload-processing" Nov 01 06:51:42.195670 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.195606 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5dec640-0c5c-483a-b87f-6aba894a88cc" containerName="payload-processing" Nov 01 06:51:42.195708 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.195692 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5dec640-0c5c-483a-b87f-6aba894a88cc" containerName="payload-processing" Nov 01 06:51:42.200371 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.200352 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb" Nov 01 06:51:42.287796 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.287760 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f1c27f-e62f-42ed-8633-829b8b66e8d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb\" (UID: \"b0f1c27f-e62f-42ed-8633-829b8b66e8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb" Nov 01 06:51:42.288000 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.287860 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hthj\" (UniqueName: \"kubernetes.io/projected/b0f1c27f-e62f-42ed-8633-829b8b66e8d0-kube-api-access-4hthj\") pod \"payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb\" (UID: \"b0f1c27f-e62f-42ed-8633-829b8b66e8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb" Nov 01 06:51:42.303482 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.303443 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-28c598\"" Nov 01 06:51:42.303687 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.303665 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-28c598-dockercfg-gc9r2\"" Nov 01 06:51:42.307537 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.307513 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb"] Nov 01 06:51:42.388869 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.388782 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hthj\" (UniqueName: \"kubernetes.io/projected/b0f1c27f-e62f-42ed-8633-829b8b66e8d0-kube-api-access-4hthj\") pod \"payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb\" (UID: \"b0f1c27f-e62f-42ed-8633-829b8b66e8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb" Nov 01 06:51:42.393525 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.393481 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f1c27f-e62f-42ed-8633-829b8b66e8d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb\" (UID: \"b0f1c27f-e62f-42ed-8633-829b8b66e8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb" Nov 01 06:51:42.394473 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.394443 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f1c27f-e62f-42ed-8633-829b8b66e8d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb\" (UID: \"b0f1c27f-e62f-42ed-8633-829b8b66e8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb" Nov 01 06:51:42.396670 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.396645 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5dec640-0c5c-483a-b87f-6aba894a88cc" path="/var/lib/kubelet/pods/f5dec640-0c5c-483a-b87f-6aba894a88cc/volumes" Nov 01 06:51:42.397235 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.397218 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2"] Nov 01 06:51:42.403797 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.402410 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2" Nov 01 06:51:42.405755 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.405729 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2"] Nov 01 06:51:42.446044 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.445972 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hthj\" (UniqueName: \"kubernetes.io/projected/b0f1c27f-e62f-42ed-8633-829b8b66e8d0-kube-api-access-4hthj\") pod \"payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb\" (UID: \"b0f1c27f-e62f-42ed-8633-829b8b66e8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb" Nov 01 06:51:42.494924 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.494884 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfxl4\" (UniqueName: \"kubernetes.io/projected/d00aff18-fddd-4938-aa74-87917e5a721d-kube-api-access-hfxl4\") pod \"payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2\" (UID: \"d00aff18-fddd-4938-aa74-87917e5a721d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2" Nov 01 06:51:42.495106 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.494938 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d00aff18-fddd-4938-aa74-87917e5a721d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2\" (UID: \"d00aff18-fddd-4938-aa74-87917e5a721d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2" Nov 01 06:51:42.514101 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.514074 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb" Nov 01 06:51:42.596076 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.596039 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfxl4\" (UniqueName: \"kubernetes.io/projected/d00aff18-fddd-4938-aa74-87917e5a721d-kube-api-access-hfxl4\") pod \"payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2\" (UID: \"d00aff18-fddd-4938-aa74-87917e5a721d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2" Nov 01 06:51:42.596235 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.596111 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d00aff18-fddd-4938-aa74-87917e5a721d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2\" (UID: \"d00aff18-fddd-4938-aa74-87917e5a721d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2" Nov 01 06:51:42.597043 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.597021 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d00aff18-fddd-4938-aa74-87917e5a721d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2\" (UID: \"d00aff18-fddd-4938-aa74-87917e5a721d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2" Nov 01 06:51:42.652392 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.652356 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfxl4\" (UniqueName: \"kubernetes.io/projected/d00aff18-fddd-4938-aa74-87917e5a721d-kube-api-access-hfxl4\") pod \"payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2\" (UID: \"d00aff18-fddd-4938-aa74-87917e5a721d\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2" Nov 01 06:51:42.722702 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.722609 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2" Nov 01 06:51:42.889685 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:42.889656 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb"] Nov 01 06:51:42.890404 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:51:42.890380 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f1c27f_e62f_42ed_8633_829b8b66e8d0.slice/crio-bd34c950abea8e1864c1db6855c17b32b8a1c048c270ff3743fb2e6ec7ea5b46 WatchSource:0}: Error finding container bd34c950abea8e1864c1db6855c17b32b8a1c048c270ff3743fb2e6ec7ea5b46: Status 404 returned error can't find the container with id bd34c950abea8e1864c1db6855c17b32b8a1c048c270ff3743fb2e6ec7ea5b46 Nov 01 06:51:43.101349 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:43.101320 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2"] Nov 01 06:51:43.101813 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:51:43.101783 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00aff18_fddd_4938_aa74_87917e5a721d.slice/crio-77099b7ac559989c62666c445f99904e6d353a19e1b2b49bf6b64c951a60cac5 WatchSource:0}: Error finding container 77099b7ac559989c62666c445f99904e6d353a19e1b2b49bf6b64c951a60cac5: Status 404 returned error can't find the container with id 77099b7ac559989c62666c445f99904e6d353a19e1b2b49bf6b64c951a60cac5 Nov 01 06:51:43.185291 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:43.185247 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb" event={"ID":"b0f1c27f-e62f-42ed-8633-829b8b66e8d0","Type":"ContainerStarted","Data":"1c7f5d387e902703b205d57f79de606e0115a9e251bce9c4be21057e30195f55"} Nov 01 06:51:43.185450 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:43.185326 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb" event={"ID":"b0f1c27f-e62f-42ed-8633-829b8b66e8d0","Type":"ContainerStarted","Data":"bd34c950abea8e1864c1db6855c17b32b8a1c048c270ff3743fb2e6ec7ea5b46"} Nov 01 06:51:43.185450 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:43.185415 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb" Nov 01 06:51:43.187193 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:43.187165 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2" event={"ID":"d00aff18-fddd-4938-aa74-87917e5a721d","Type":"ContainerStarted","Data":"16acb300fa4028f4d4a214ac77c3831ed98c8f669b0de1666d9264c01c4aeab7"} Nov 01 06:51:43.187315 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:43.187217 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2" event={"ID":"d00aff18-fddd-4938-aa74-87917e5a721d","Type":"ContainerStarted","Data":"77099b7ac559989c62666c445f99904e6d353a19e1b2b49bf6b64c951a60cac5"} Nov 01 06:51:43.187315 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:43.187267 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2" Nov 01 06:51:43.301634 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:43.301572 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb" podStartSLOduration=1.301553018 podStartE2EDuration="1.301553018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:51:43.297462233 +0000 UTC m=+2513.061262091" watchObservedRunningTime="2026-11-01 06:51:43.301553018 +0000 UTC m=+2513.065352876" Nov 01 06:51:43.413977 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:43.413862 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2" podStartSLOduration=1.413846207 podStartE2EDuration="1.413846207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:51:43.409564428 +0000 UTC m=+2513.173364286" watchObservedRunningTime="2026-11-01 06:51:43.413846207 +0000 UTC m=+2513.177646061" Nov 01 06:51:43.811477 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:43.811375 4385 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0121f247d8fb: secret "agc-cred-0121f247d8fb" not found Nov 01 06:51:43.811477 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:43.811412 4385 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4: secret "agc-cred-0121f247d8fb" not found Nov 01 06:51:43.811477 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:51:43.811462 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-provider-credentials podName:73f499a0-aa4a-4a99-af16-2d3775701b21 nodeName:}" failed. No retries permitted until 2026-11-01 06:51:51.811448067 +0000 UTC m=+2521.575247903 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-provider-credentials") pod "payload-processing-external-model-6bd55dd7d7-7swd4" (UID: "73f499a0-aa4a-4a99-af16-2d3775701b21") : secret "agc-cred-0121f247d8fb" not found Nov 01 06:51:43.874869 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:43.874811 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:44.023862 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:44.023806 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" podUID="73f499a0-aa4a-4a99-af16-2d3775701b21" containerName="payload-processing" containerID="cri-o://a954d50d6221a8b19b0da6d3eda78586054cc3aa1bf2c52beabb5012d6c0ee0c" gracePeriod=25 Nov 01 06:51:44.072055 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:44.071952 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:44.203301 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:44.197980 4385 generic.go:356] "Generic (PLEG): container finished" podID="73f499a0-aa4a-4a99-af16-2d3775701b21" containerID="a954d50d6221a8b19b0da6d3eda78586054cc3aa1bf2c52beabb5012d6c0ee0c" exitCode=0 Nov 01 06:51:44.203301 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:44.199300 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" event={"ID":"73f499a0-aa4a-4a99-af16-2d3775701b21","Type":"ContainerDied","Data":"a954d50d6221a8b19b0da6d3eda78586054cc3aa1bf2c52beabb5012d6c0ee0c"} Nov 01 06:51:44.302367 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:44.302335 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" Nov 01 06:51:44.318449 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:44.318406 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-kube-api-access-x85sl\" (UniqueName: \"kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-kube-api-access-x85sl\") pod \"73f499a0-aa4a-4a99-af16-2d3775701b21\" (UID: \"73f499a0-aa4a-4a99-af16-2d3775701b21\") " Nov 01 06:51:44.318638 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:44.318504 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73f499a0-aa4a-4a99-af16-2d3775701b21-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/73f499a0-aa4a-4a99-af16-2d3775701b21-routing-overlay-volume\") pod \"73f499a0-aa4a-4a99-af16-2d3775701b21\" (UID: \"73f499a0-aa4a-4a99-af16-2d3775701b21\") " Nov 01 06:51:44.318638 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:44.318550 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-provider-credentials\" (UniqueName: \"kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-provider-credentials\") pod \"73f499a0-aa4a-4a99-af16-2d3775701b21\" (UID: \"73f499a0-aa4a-4a99-af16-2d3775701b21\") " Nov 01 06:51:44.318751 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:44.318701 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73f499a0-aa4a-4a99-af16-2d3775701b21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f499a0-aa4a-4a99-af16-2d3775701b21-plugins-config-volume\") pod \"73f499a0-aa4a-4a99-af16-2d3775701b21\" (UID: \"73f499a0-aa4a-4a99-af16-2d3775701b21\") " Nov 01 06:51:44.318916 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:44.318884 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f499a0-aa4a-4a99-af16-2d3775701b21-routing-overlay-volume" pod "73f499a0-aa4a-4a99-af16-2d3775701b21" (UID: "73f499a0-aa4a-4a99-af16-2d3775701b21"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:51:44.319176 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:44.319117 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f499a0-aa4a-4a99-af16-2d3775701b21-plugins-config-volume" pod "73f499a0-aa4a-4a99-af16-2d3775701b21" (UID: "73f499a0-aa4a-4a99-af16-2d3775701b21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:51:44.320896 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:44.320865 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-kube-api-access-x85sl" pod "73f499a0-aa4a-4a99-af16-2d3775701b21" (UID: "73f499a0-aa4a-4a99-af16-2d3775701b21"). InnerVolumeSpecName "kube-api-access-x85sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:51:44.320978 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:44.320893 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-provider-credentials" pod "73f499a0-aa4a-4a99-af16-2d3775701b21" (UID: "73f499a0-aa4a-4a99-af16-2d3775701b21"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:51:44.420871 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:44.420819 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f499a0-aa4a-4a99-af16-2d3775701b21-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:44.420871 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:44.420870 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x85sl\" (UniqueName: \"kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-kube-api-access-x85sl\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:44.420871 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:44.420884 4385 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/73f499a0-aa4a-4a99-af16-2d3775701b21-routing-overlay-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:44.421349 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:44.420899 4385 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/73f499a0-aa4a-4a99-af16-2d3775701b21-provider-credentials\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:45.207742 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:45.207706 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" Nov 01 06:51:45.208061 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:45.207732 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4" event={"ID":"73f499a0-aa4a-4a99-af16-2d3775701b21","Type":"ContainerDied","Data":"ecee5ba2b0639df1e7eb35861072ffbd4af6c43d32f15a23f246cd0c3f0825a0"} Nov 01 06:51:45.208061 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:45.207821 4385 scope.go:122] "RemoveContainer" containerID="a954d50d6221a8b19b0da6d3eda78586054cc3aa1bf2c52beabb5012d6c0ee0c" Nov 01 06:51:45.372794 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:45.372761 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4"] Nov 01 06:51:45.421908 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:45.421870 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd55dd7d7-7swd4"] Nov 01 06:51:46.387232 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:46.387202 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73f499a0-aa4a-4a99-af16-2d3775701b21" path="/var/lib/kubelet/pods/73f499a0-aa4a-4a99-af16-2d3775701b21/volumes" Nov 01 06:51:46.524223 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:46.524194 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:48.083451 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:48.083414 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:51.171933 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:51.171889 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:51:51.359516 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:51.359477 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:51.555514 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:51.555460 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:51.709084 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:51.708976 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:51:51.750256 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:51.750222 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng"] Nov 01 06:51:51.750724 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:51.750712 4385 cpu_manager.go:414] "containerMap: removing container" podUID="73f499a0-aa4a-4a99-af16-2d3775701b21" containerName="payload-processing" Nov 01 06:51:51.750767 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:51.750725 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f499a0-aa4a-4a99-af16-2d3775701b21" containerName="payload-processing" Nov 01 06:51:51.750864 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:51.750847 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f499a0-aa4a-4a99-af16-2d3775701b21" containerName="payload-processing" Nov 01 06:51:51.755476 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:51.755460 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng" Nov 01 06:51:51.810888 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:51.810855 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng"] Nov 01 06:51:51.815739 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:51.815704 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt4fg\" (UniqueName: \"kubernetes.io/projected/15b0a39a-6939-433a-803a-8cdaf6dac909-kube-api-access-lt4fg\") pod \"payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng\" (UID: \"15b0a39a-6939-433a-803a-8cdaf6dac909\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng" Nov 01 06:51:51.816001 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:51.815981 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15b0a39a-6939-433a-803a-8cdaf6dac909-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng\" (UID: \"15b0a39a-6939-433a-803a-8cdaf6dac909\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng" Nov 01 06:51:51.856398 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:51.856364 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-a3e841\"" Nov 01 06:51:51.856668 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:51.856527 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-a3e841-dockercfg-nfgxw\"" Nov 01 06:51:51.916963 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:51.916920 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt4fg\" (UniqueName: \"kubernetes.io/projected/15b0a39a-6939-433a-803a-8cdaf6dac909-kube-api-access-lt4fg\") pod \"payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng\" (UID: \"15b0a39a-6939-433a-803a-8cdaf6dac909\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng" Nov 01 06:51:51.917181 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:51.917004 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15b0a39a-6939-433a-803a-8cdaf6dac909-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng\" (UID: \"15b0a39a-6939-433a-803a-8cdaf6dac909\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng" Nov 01 06:51:51.918199 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:51.918171 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15b0a39a-6939-433a-803a-8cdaf6dac909-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng\" (UID: \"15b0a39a-6939-433a-803a-8cdaf6dac909\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng" Nov 01 06:51:51.980283 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:51.980199 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt4fg\" (UniqueName: \"kubernetes.io/projected/15b0a39a-6939-433a-803a-8cdaf6dac909-kube-api-access-lt4fg\") pod \"payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng\" (UID: \"15b0a39a-6939-433a-803a-8cdaf6dac909\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng" Nov 01 06:51:52.069046 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:52.069005 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng" Nov 01 06:51:52.444804 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:52.444775 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng"] Nov 01 06:51:52.445425 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:51:52.445399 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b0a39a_6939_433a_803a_8cdaf6dac909.slice/crio-8edc56e3f42f8a4efa46e2d76df2aa11f47e2727ac090713ae1324ea43da71b5 WatchSource:0}: Error finding container 8edc56e3f42f8a4efa46e2d76df2aa11f47e2727ac090713ae1324ea43da71b5: Status 404 returned error can't find the container with id 8edc56e3f42f8a4efa46e2d76df2aa11f47e2727ac090713ae1324ea43da71b5 Nov 01 06:51:53.272454 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:53.272411 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng" event={"ID":"15b0a39a-6939-433a-803a-8cdaf6dac909","Type":"ContainerStarted","Data":"b7c9ca05def068f3a29b90c63ad78dc5a14e6532330b56744236ae7dc59d81b0"} Nov 01 06:51:53.272668 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:53.272508 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng" event={"ID":"15b0a39a-6939-433a-803a-8cdaf6dac909","Type":"ContainerStarted","Data":"8edc56e3f42f8a4efa46e2d76df2aa11f47e2727ac090713ae1324ea43da71b5"} Nov 01 06:51:53.272795 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:53.272661 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng" Nov 01 06:51:53.387614 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:53.387562 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng" podStartSLOduration=2.387546254 podStartE2EDuration="2.387546254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:51:53.384697484 +0000 UTC m=+2523.148497356" watchObservedRunningTime="2026-11-01 06:51:53.387546254 +0000 UTC m=+2523.151346110" Nov 01 06:51:54.202427 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:54.202398 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2" Nov 01 06:51:54.202957 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:54.202816 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb" Nov 01 06:51:56.806695 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:51:56.806663 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:01.169861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:01.169799 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:52:01.617158 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:01.617124 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:04.287032 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:04.287003 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng" Nov 01 06:52:06.315560 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:06.315527 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:06.783984 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:52:06.783944 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:52:10.947527 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:10.947496 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:11.170119 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:11.170086 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:52:15.581372 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:15.581297 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:20.215533 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:20.215491 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:21.170231 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:21.170193 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:52:24.920729 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:24.920691 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:29.617796 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:29.617760 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:31.170474 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:31.170425 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:52:33.946372 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:33.946334 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:34.140522 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:34.140484 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:34.332751 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:34.332721 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:36.779190 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:52:36.779154 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:52:36.852398 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:36.852364 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:37.053140 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:37.053053 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:38.781500 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:38.781471 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9"] Nov 01 06:52:38.785499 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:38.785482 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" Nov 01 06:52:38.838493 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:38.838467 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-28c598-serving-cert\"" Nov 01 06:52:38.847553 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:38.846290 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9"] Nov 01 06:52:38.939243 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:38.939208 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bfe801-8222-497c-85ce-1fc5695f32e7-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9\" (UID: \"c2bfe801-8222-497c-85ce-1fc5695f32e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" Nov 01 06:52:38.939399 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:38.939295 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2bfe801-8222-497c-85ce-1fc5695f32e7-metrics-certs\") pod \"maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9\" (UID: \"c2bfe801-8222-497c-85ce-1fc5695f32e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" Nov 01 06:52:38.939399 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:38.939337 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dgvc\" (UniqueName: \"kubernetes.io/projected/c2bfe801-8222-497c-85ce-1fc5695f32e7-kube-api-access-5dgvc\") pod \"maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9\" (UID: \"c2bfe801-8222-497c-85ce-1fc5695f32e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" Nov 01 06:52:38.939471 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:38.939441 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2bfe801-8222-497c-85ce-1fc5695f32e7-maas-api-tls\") pod \"maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9\" (UID: \"c2bfe801-8222-497c-85ce-1fc5695f32e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" Nov 01 06:52:38.983913 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:38.983876 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:39.041099 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:39.041024 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bfe801-8222-497c-85ce-1fc5695f32e7-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9\" (UID: \"c2bfe801-8222-497c-85ce-1fc5695f32e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" Nov 01 06:52:39.041099 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:39.041073 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2bfe801-8222-497c-85ce-1fc5695f32e7-metrics-certs\") pod \"maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9\" (UID: \"c2bfe801-8222-497c-85ce-1fc5695f32e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" Nov 01 06:52:39.041310 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:39.041115 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dgvc\" (UniqueName: \"kubernetes.io/projected/c2bfe801-8222-497c-85ce-1fc5695f32e7-kube-api-access-5dgvc\") pod \"maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9\" (UID: \"c2bfe801-8222-497c-85ce-1fc5695f32e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" Nov 01 06:52:39.041310 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:39.041193 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2bfe801-8222-497c-85ce-1fc5695f32e7-maas-api-tls\") pod \"maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9\" (UID: \"c2bfe801-8222-497c-85ce-1fc5695f32e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" Nov 01 06:52:39.041753 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:39.041728 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bfe801-8222-497c-85ce-1fc5695f32e7-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9\" (UID: \"c2bfe801-8222-497c-85ce-1fc5695f32e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" Nov 01 06:52:39.043694 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:39.043674 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2bfe801-8222-497c-85ce-1fc5695f32e7-metrics-certs\") pod \"maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9\" (UID: \"c2bfe801-8222-497c-85ce-1fc5695f32e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" Nov 01 06:52:39.043809 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:39.043711 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2bfe801-8222-497c-85ce-1fc5695f32e7-maas-api-tls\") pod \"maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9\" (UID: \"c2bfe801-8222-497c-85ce-1fc5695f32e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" Nov 01 06:52:39.098370 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:39.098339 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dgvc\" (UniqueName: \"kubernetes.io/projected/c2bfe801-8222-497c-85ce-1fc5695f32e7-kube-api-access-5dgvc\") pod \"maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9\" (UID: \"c2bfe801-8222-497c-85ce-1fc5695f32e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" Nov 01 06:52:39.098544 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:39.098524 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" Nov 01 06:52:39.186029 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:39.185868 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:39.329238 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:39.329210 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:39.474120 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:52:39.474091 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2bfe801_8222_497c_85ce_1fc5695f32e7.slice/crio-9da0e001c407c5451da0d843929adc726b8b69bc30f3aa01b55fcdec1b4bcadb WatchSource:0}: Error finding container 9da0e001c407c5451da0d843929adc726b8b69bc30f3aa01b55fcdec1b4bcadb: Status 404 returned error can't find the container with id 9da0e001c407c5451da0d843929adc726b8b69bc30f3aa01b55fcdec1b4bcadb Nov 01 06:52:39.474254 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:39.474230 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9"] Nov 01 06:52:39.475783 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:39.475768 4385 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:52:39.594519 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:39.594449 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" event={"ID":"c2bfe801-8222-497c-85ce-1fc5695f32e7","Type":"ContainerStarted","Data":"9da0e001c407c5451da0d843929adc726b8b69bc30f3aa01b55fcdec1b4bcadb"} Nov 01 06:52:40.603723 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:40.603687 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" event={"ID":"c2bfe801-8222-497c-85ce-1fc5695f32e7","Type":"ContainerStarted","Data":"a747680a3558aa280511876d75e87c5084ebf57c6a62e1c232a8d40bd606ec5e"} Nov 01 06:52:40.604117 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:40.603747 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" Nov 01 06:52:40.625021 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:40.624988 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:40.720895 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:40.720845 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" podStartSLOduration=1.8949836119999999 podStartE2EDuration="2.720818982s" totalImagesPullingTime="825.83537ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:52:40.718512088 +0000 UTC m=+2570.482311943" watchObservedRunningTime="2026-11-01 06:52:40.720818982 +0000 UTC m=+2570.484618838" Nov 01 06:52:41.069574 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:41.069493 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:41.169755 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:41.169715 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:52:41.713042 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:41.713012 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:41.996585 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:41.996502 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:44.424372 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:44.424341 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:46.304460 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:46.304421 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:46.474040 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:46.473994 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:46.623677 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:46.623634 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:46.683041 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:46.683011 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" Nov 01 06:52:48.424046 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:48.424016 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:49.527341 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:49.527303 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:51.170501 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:51.170460 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:52:51.910726 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:51.910692 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:52.100335 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:52.100297 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:53.266536 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:53.266493 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:53.437217 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:53.437174 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:53.720208 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:53.720169 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:53.823603 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:53.823563 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5"] Nov 01 06:52:53.828575 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:53.828542 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5" Nov 01 06:52:53.869149 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:53.869116 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:53.894937 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:53.894905 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5"] Nov 01 06:52:53.933116 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:53.933079 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b77kw\" (UniqueName: \"kubernetes.io/projected/1a10a1d9-d645-42fe-9aa1-d79a35398d68-kube-api-access-b77kw\") pod \"payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5\" (UID: \"1a10a1d9-d645-42fe-9aa1-d79a35398d68\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5" Nov 01 06:52:53.933271 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:53.933137 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a10a1d9-d645-42fe-9aa1-d79a35398d68-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5\" (UID: \"1a10a1d9-d645-42fe-9aa1-d79a35398d68\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5" Nov 01 06:52:53.937447 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:53.937415 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-6ba6e5\"" Nov 01 06:52:53.937585 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:53.937472 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-6ba6e5-dockercfg-ktf5p\"" Nov 01 06:52:53.941009 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:53.940985 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58"] Nov 01 06:52:53.944818 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:53.944801 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" Nov 01 06:52:53.971106 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:53.971050 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58"] Nov 01 06:52:53.997371 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:53.997342 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-a3e841-serving-cert\"" Nov 01 06:52:54.034316 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.034279 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-openshift-service-ca\") pod \"maas-api-e2e-autores-a3e841-849cdc6685-9wc58\" (UID: \"7efc28e3-9824-4c6b-ae7e-e9323ee6a130\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" Nov 01 06:52:54.034457 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.034331 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-metrics-certs\") pod \"maas-api-e2e-autores-a3e841-849cdc6685-9wc58\" (UID: \"7efc28e3-9824-4c6b-ae7e-e9323ee6a130\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" Nov 01 06:52:54.034457 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.034402 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b869d\" (UniqueName: \"kubernetes.io/projected/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-kube-api-access-b869d\") pod \"maas-api-e2e-autores-a3e841-849cdc6685-9wc58\" (UID: \"7efc28e3-9824-4c6b-ae7e-e9323ee6a130\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" Nov 01 06:52:54.034535 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.034518 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b77kw\" (UniqueName: \"kubernetes.io/projected/1a10a1d9-d645-42fe-9aa1-d79a35398d68-kube-api-access-b77kw\") pod \"payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5\" (UID: \"1a10a1d9-d645-42fe-9aa1-d79a35398d68\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5" Nov 01 06:52:54.034569 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.034557 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a10a1d9-d645-42fe-9aa1-d79a35398d68-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5\" (UID: \"1a10a1d9-d645-42fe-9aa1-d79a35398d68\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5" Nov 01 06:52:54.034602 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.034578 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-maas-api-tls\") pod \"maas-api-e2e-autores-a3e841-849cdc6685-9wc58\" (UID: \"7efc28e3-9824-4c6b-ae7e-e9323ee6a130\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" Nov 01 06:52:54.035245 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.035226 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a10a1d9-d645-42fe-9aa1-d79a35398d68-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5\" (UID: \"1a10a1d9-d645-42fe-9aa1-d79a35398d68\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5" Nov 01 06:52:54.092317 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.092280 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b77kw\" (UniqueName: \"kubernetes.io/projected/1a10a1d9-d645-42fe-9aa1-d79a35398d68-kube-api-access-b77kw\") pod \"payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5\" (UID: \"1a10a1d9-d645-42fe-9aa1-d79a35398d68\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5" Nov 01 06:52:54.132597 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.132562 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:54.136195 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.136164 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-maas-api-tls\") pod \"maas-api-e2e-autores-a3e841-849cdc6685-9wc58\" (UID: \"7efc28e3-9824-4c6b-ae7e-e9323ee6a130\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" Nov 01 06:52:54.136353 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.136250 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-openshift-service-ca\") pod \"maas-api-e2e-autores-a3e841-849cdc6685-9wc58\" (UID: \"7efc28e3-9824-4c6b-ae7e-e9323ee6a130\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" Nov 01 06:52:54.136353 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.136309 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-metrics-certs\") pod \"maas-api-e2e-autores-a3e841-849cdc6685-9wc58\" (UID: \"7efc28e3-9824-4c6b-ae7e-e9323ee6a130\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" Nov 01 06:52:54.136353 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:52:54.136313 4385 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-serving-cert: secret "maas-api-e2e-autores-a3e841-serving-cert" not found Nov 01 06:52:54.136353 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.136346 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b869d\" (UniqueName: \"kubernetes.io/projected/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-kube-api-access-b869d\") pod \"maas-api-e2e-autores-a3e841-849cdc6685-9wc58\" (UID: \"7efc28e3-9824-4c6b-ae7e-e9323ee6a130\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" Nov 01 06:52:54.136560 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:52:54.136374 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-maas-api-tls podName:7efc28e3-9824-4c6b-ae7e-e9323ee6a130 nodeName:}" failed. No retries permitted until 2026-11-01 06:52:54.636355541 +0000 UTC m=+2584.400155393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-maas-api-tls") pod "maas-api-e2e-autores-a3e841-849cdc6685-9wc58" (UID: "7efc28e3-9824-4c6b-ae7e-e9323ee6a130") : secret "maas-api-e2e-autores-a3e841-serving-cert" not found Nov 01 06:52:54.136993 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.136965 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-openshift-service-ca\") pod \"maas-api-e2e-autores-a3e841-849cdc6685-9wc58\" (UID: \"7efc28e3-9824-4c6b-ae7e-e9323ee6a130\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" Nov 01 06:52:54.138971 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.138943 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-metrics-certs\") pod \"maas-api-e2e-autores-a3e841-849cdc6685-9wc58\" (UID: \"7efc28e3-9824-4c6b-ae7e-e9323ee6a130\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" Nov 01 06:52:54.145882 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.145863 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5" Nov 01 06:52:54.195876 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.194081 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b869d\" (UniqueName: \"kubernetes.io/projected/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-kube-api-access-b869d\") pod \"maas-api-e2e-autores-a3e841-849cdc6685-9wc58\" (UID: \"7efc28e3-9824-4c6b-ae7e-e9323ee6a130\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" Nov 01 06:52:54.283402 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.283367 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:54.435033 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.434999 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:52:54.541365 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:52:54.541332 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a10a1d9_d645_42fe_9aa1_d79a35398d68.slice/crio-181e07b26d66f2ffa7ca696b4a6b0f88812eb9076d428d7468fbf6677bb4f34b WatchSource:0}: Error finding container 181e07b26d66f2ffa7ca696b4a6b0f88812eb9076d428d7468fbf6677bb4f34b: Status 404 returned error can't find the container with id 181e07b26d66f2ffa7ca696b4a6b0f88812eb9076d428d7468fbf6677bb4f34b Nov 01 06:52:54.541511 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.541477 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5"] Nov 01 06:52:54.642268 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.642242 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-maas-api-tls\") pod \"maas-api-e2e-autores-a3e841-849cdc6685-9wc58\" (UID: \"7efc28e3-9824-4c6b-ae7e-e9323ee6a130\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" Nov 01 06:52:54.644537 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.644516 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-maas-api-tls\") pod \"maas-api-e2e-autores-a3e841-849cdc6685-9wc58\" (UID: \"7efc28e3-9824-4c6b-ae7e-e9323ee6a130\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" Nov 01 06:52:54.760169 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.760137 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5" event={"ID":"1a10a1d9-d645-42fe-9aa1-d79a35398d68","Type":"ContainerStarted","Data":"edd9dbc5f7bad91f6439344fec9bdb251df1b6c5f2e21a708eefa065ffbe3c2e"} Nov 01 06:52:54.760371 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.760188 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5" event={"ID":"1a10a1d9-d645-42fe-9aa1-d79a35398d68","Type":"ContainerStarted","Data":"181e07b26d66f2ffa7ca696b4a6b0f88812eb9076d428d7468fbf6677bb4f34b"} Nov 01 06:52:54.760371 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.760263 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5" Nov 01 06:52:54.858226 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.858199 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" Nov 01 06:52:54.878411 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:54.878367 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5" podStartSLOduration=1.878353986 podStartE2EDuration="1.878353986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:52:54.875052143 +0000 UTC m=+2584.638851999" watchObservedRunningTime="2026-11-01 06:52:54.878353986 +0000 UTC m=+2584.642153863" Nov 01 06:52:55.283878 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:55.282526 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58"] Nov 01 06:52:55.284349 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:52:55.283972 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7efc28e3_9824_4c6b_ae7e_e9323ee6a130.slice/crio-6647b900d3394b42aad3fa2e5b7b83cab070a89c1d28b2faabea8cceb4f58304 WatchSource:0}: Error finding container 6647b900d3394b42aad3fa2e5b7b83cab070a89c1d28b2faabea8cceb4f58304: Status 404 returned error can't find the container with id 6647b900d3394b42aad3fa2e5b7b83cab070a89c1d28b2faabea8cceb4f58304 Nov 01 06:52:55.767704 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:55.767670 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" event={"ID":"7efc28e3-9824-4c6b-ae7e-e9323ee6a130","Type":"ContainerStarted","Data":"6647b900d3394b42aad3fa2e5b7b83cab070a89c1d28b2faabea8cceb4f58304"} Nov 01 06:52:56.775595 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:56.775564 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" event={"ID":"7efc28e3-9824-4c6b-ae7e-e9323ee6a130","Type":"ContainerStarted","Data":"d4e34b3167a3166dab408adefa0ae5bc6d5a1f899a2f411d6e81f27260e9cf0d"} Nov 01 06:52:56.776025 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:56.775733 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" Nov 01 06:52:58.950947 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:52:58.950907 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:53:01.170053 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:01.170010 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:53:01.378294 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:01.378257 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:53:02.788618 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:02.788591 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" Nov 01 06:53:02.903949 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:02.903896 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" podStartSLOduration=9.119541635 podStartE2EDuration="9.903883111s" totalImagesPullingTime="784.341476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:52:56.892033529 +0000 UTC m=+2586.655833380" watchObservedRunningTime="2026-11-01 06:53:02.903883111 +0000 UTC m=+2592.667683030" Nov 01 06:53:03.632556 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:03.632522 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:53:05.770377 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:05.770345 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5" Nov 01 06:53:06.780622 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:53:06.780580 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:53:08.333550 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:08.333511 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:53:11.378755 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:11.378723 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:53:13.209727 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:13.209689 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:53:18.045292 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:18.045250 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:53:21.378395 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:21.378362 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:53:22.755579 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:22.755538 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:53:22.953234 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:22.953197 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:53:27.638814 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:27.638777 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:53:31.378532 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:31.378496 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:53:32.321546 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:32.321512 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:53:32.653488 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:32.653460 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:53:36.779073 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:53:36.779035 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:53:36.954662 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:36.954630 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:53:41.378237 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:41.378201 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:53:41.581002 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:41.580961 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:53:46.335484 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:46.335442 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:53:48.754327 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:48.754290 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:53:51.023089 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:51.023056 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:53:51.378531 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:51.378497 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:53:55.812959 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:55.812921 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:53:58.574409 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:58.574374 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:53:59.106439 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:59.106402 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:53:59.262634 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:53:59.262598 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:00.769298 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:00.769264 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:00.928968 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:00.928927 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:01.076476 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:01.076447 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:01.378377 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:01.378270 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:54:03.414060 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:03.414017 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:05.623532 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:05.623494 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:06.778541 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:54:06.778508 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:54:10.304793 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:10.304758 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:11.378731 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:11.378699 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:54:12.896620 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:12.896582 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:13.109136 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:13.109103 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:15.002170 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:15.002128 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:16.386850 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:16.386799 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:54:17.241731 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:17.241689 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:20.003963 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:20.003927 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:24.658866 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:24.658817 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:26.387450 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:26.387413 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:54:29.432800 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:29.432764 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:34.164822 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:34.164792 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:36.387603 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:36.387563 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:54:36.753020 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:36.752926 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:36.778879 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:54:36.778813 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:54:36.891640 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:36.891607 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:37.716690 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:37.716661 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q"] Nov 01 06:54:37.717975 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:37.717952 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:54:38.912388 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:38.912342 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:42.717132 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:42.717081 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" containerID="cri-o://91b985cf70b13f002b4ab8f7f2fe1839a5de57e0d6541366b3322e67e117f3b2" gracePeriod=25 Nov 01 06:54:42.978796 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:42.978738 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" Nov 01 06:54:43.067075 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.067047 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b0f033-c464-4bab-b201-07686045040e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71b0f033-c464-4bab-b201-07686045040e-routing-overlay-volume\") pod \"71b0f033-c464-4bab-b201-07686045040e\" (UID: \"71b0f033-c464-4bab-b201-07686045040e\") " Nov 01 06:54:43.067220 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.067158 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b0f033-c464-4bab-b201-07686045040e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b0f033-c464-4bab-b201-07686045040e-plugins-config-volume\") pod \"71b0f033-c464-4bab-b201-07686045040e\" (UID: \"71b0f033-c464-4bab-b201-07686045040e\") " Nov 01 06:54:43.067308 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.067292 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b0f033-c464-4bab-b201-07686045040e-kube-api-access-n5p9n\" (UniqueName: \"kubernetes.io/projected/71b0f033-c464-4bab-b201-07686045040e-kube-api-access-n5p9n\") pod \"71b0f033-c464-4bab-b201-07686045040e\" (UID: \"71b0f033-c464-4bab-b201-07686045040e\") " Nov 01 06:54:43.067407 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.067391 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b0f033-c464-4bab-b201-07686045040e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/71b0f033-c464-4bab-b201-07686045040e-provider-credentials\") pod \"71b0f033-c464-4bab-b201-07686045040e\" (UID: \"71b0f033-c464-4bab-b201-07686045040e\") " Nov 01 06:54:43.067479 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.067436 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b0f033-c464-4bab-b201-07686045040e-routing-overlay-volume" pod "71b0f033-c464-4bab-b201-07686045040e" (UID: "71b0f033-c464-4bab-b201-07686045040e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:54:43.067561 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.067536 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b0f033-c464-4bab-b201-07686045040e-plugins-config-volume" pod "71b0f033-c464-4bab-b201-07686045040e" (UID: "71b0f033-c464-4bab-b201-07686045040e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:54:43.067783 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.067766 4385 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/71b0f033-c464-4bab-b201-07686045040e-routing-overlay-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:54:43.067880 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.067785 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b0f033-c464-4bab-b201-07686045040e-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:54:43.069471 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.069449 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b0f033-c464-4bab-b201-07686045040e-kube-api-access-n5p9n" pod "71b0f033-c464-4bab-b201-07686045040e" (UID: "71b0f033-c464-4bab-b201-07686045040e"). InnerVolumeSpecName "kube-api-access-n5p9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:54:43.069557 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.069496 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b0f033-c464-4bab-b201-07686045040e-provider-credentials" pod "71b0f033-c464-4bab-b201-07686045040e" (UID: "71b0f033-c464-4bab-b201-07686045040e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:54:43.168779 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.168746 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5p9n\" (UniqueName: \"kubernetes.io/projected/71b0f033-c464-4bab-b201-07686045040e-kube-api-access-n5p9n\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:54:43.168779 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.168772 4385 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/71b0f033-c464-4bab-b201-07686045040e-provider-credentials\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:54:43.488297 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.488259 4385 generic.go:356] "Generic (PLEG): container finished" podID="71b0f033-c464-4bab-b201-07686045040e" containerID="91b985cf70b13f002b4ab8f7f2fe1839a5de57e0d6541366b3322e67e117f3b2" exitCode=0 Nov 01 06:54:43.488494 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.488310 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" Nov 01 06:54:43.488494 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.488338 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" event={"ID":"71b0f033-c464-4bab-b201-07686045040e","Type":"ContainerDied","Data":"91b985cf70b13f002b4ab8f7f2fe1839a5de57e0d6541366b3322e67e117f3b2"} Nov 01 06:54:43.488494 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.488389 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q" event={"ID":"71b0f033-c464-4bab-b201-07686045040e","Type":"ContainerDied","Data":"16fe3ccf4bed3842ad80c5893565addb649a406bee6d6712d304f3b84081e10f"} Nov 01 06:54:43.488494 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.488411 4385 scope.go:122] "RemoveContainer" containerID="91b985cf70b13f002b4ab8f7f2fe1839a5de57e0d6541366b3322e67e117f3b2" Nov 01 06:54:43.501271 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.501251 4385 scope.go:122] "RemoveContainer" containerID="91b985cf70b13f002b4ab8f7f2fe1839a5de57e0d6541366b3322e67e117f3b2" Nov 01 06:54:43.501631 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:54:43.501612 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b985cf70b13f002b4ab8f7f2fe1839a5de57e0d6541366b3322e67e117f3b2\": container with ID starting with 91b985cf70b13f002b4ab8f7f2fe1839a5de57e0d6541366b3322e67e117f3b2 not found: ID does not exist" containerID="91b985cf70b13f002b4ab8f7f2fe1839a5de57e0d6541366b3322e67e117f3b2" Nov 01 06:54:43.501704 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.501639 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b985cf70b13f002b4ab8f7f2fe1839a5de57e0d6541366b3322e67e117f3b2"} err="failed to get container status \"91b985cf70b13f002b4ab8f7f2fe1839a5de57e0d6541366b3322e67e117f3b2\": rpc error: code = NotFound desc = could not find container \"91b985cf70b13f002b4ab8f7f2fe1839a5de57e0d6541366b3322e67e117f3b2\": container with ID starting with 91b985cf70b13f002b4ab8f7f2fe1839a5de57e0d6541366b3322e67e117f3b2 not found: ID does not exist" Nov 01 06:54:43.611733 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.611697 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:43.658002 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.657939 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q"] Nov 01 06:54:43.662790 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:43.662760 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-dxh9q"] Nov 01 06:54:44.385641 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:44.385608 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b0f033-c464-4bab-b201-07686045040e" path="/var/lib/kubelet/pods/71b0f033-c464-4bab-b201-07686045040e/volumes" Nov 01 06:54:46.673980 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:46.673943 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:48.358311 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:48.356440 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:49.085157 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:49.085121 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:50.156637 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:50.156539 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:50.305150 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:50.305092 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:51.328208 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:51.328163 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:51.793134 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:51.793033 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:52.215192 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:52.215158 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:53.439554 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:53.439515 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:55.213123 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:55.213086 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:54:56.322452 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.322412 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp"] Nov 01 06:54:56.323217 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.323197 4385 cpu_manager.go:414] "containerMap: removing container" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" Nov 01 06:54:56.323322 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.323219 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" Nov 01 06:54:56.323415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.323402 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b0f033-c464-4bab-b201-07686045040e" containerName="payload-processing" Nov 01 06:54:56.328098 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.328074 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" Nov 01 06:54:56.373219 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.373184 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb487cf8-3284-45ae-8809-96481a043957-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp\" (UID: \"cb487cf8-3284-45ae-8809-96481a043957\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" Nov 01 06:54:56.373344 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.373301 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb487cf8-3284-45ae-8809-96481a043957-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp\" (UID: \"cb487cf8-3284-45ae-8809-96481a043957\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" Nov 01 06:54:56.373344 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.373330 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb487cf8-3284-45ae-8809-96481a043957-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp\" (UID: \"cb487cf8-3284-45ae-8809-96481a043957\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" Nov 01 06:54:56.373461 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.373417 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5nlx\" (UniqueName: \"kubernetes.io/projected/cb487cf8-3284-45ae-8809-96481a043957-kube-api-access-d5nlx\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp\" (UID: \"cb487cf8-3284-45ae-8809-96481a043957\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" Nov 01 06:54:56.387146 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.387121 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp"] Nov 01 06:54:56.428311 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.428281 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"openshift-service-ca.crt\"" Nov 01 06:54:56.428566 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.428529 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"payload-processing-external-model-plugins-e2e-extmod-a-25719d\"" Nov 01 06:54:56.428566 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.428562 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"kube-root-ca.crt\"" Nov 01 06:54:56.428741 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.428582 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"payload-processing-external-model-e2e--bf6d8971-dockercfg-fwvkw\"" Nov 01 06:54:56.430568 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.430552 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"e2e-cmp-legacy-c0251e-provider-cred\"" Nov 01 06:54:56.430634 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.430555 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"routing-overlay\"" Nov 01 06:54:56.474425 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.474383 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb487cf8-3284-45ae-8809-96481a043957-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp\" (UID: \"cb487cf8-3284-45ae-8809-96481a043957\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" Nov 01 06:54:56.474605 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.474534 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb487cf8-3284-45ae-8809-96481a043957-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp\" (UID: \"cb487cf8-3284-45ae-8809-96481a043957\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" Nov 01 06:54:56.474605 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.474591 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb487cf8-3284-45ae-8809-96481a043957-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp\" (UID: \"cb487cf8-3284-45ae-8809-96481a043957\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" Nov 01 06:54:56.474728 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.474706 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5nlx\" (UniqueName: \"kubernetes.io/projected/cb487cf8-3284-45ae-8809-96481a043957-kube-api-access-d5nlx\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp\" (UID: \"cb487cf8-3284-45ae-8809-96481a043957\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" Nov 01 06:54:56.475576 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.475521 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb487cf8-3284-45ae-8809-96481a043957-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp\" (UID: \"cb487cf8-3284-45ae-8809-96481a043957\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" Nov 01 06:54:56.476352 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.476328 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb487cf8-3284-45ae-8809-96481a043957-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp\" (UID: \"cb487cf8-3284-45ae-8809-96481a043957\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" Nov 01 06:54:56.477911 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.477889 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb487cf8-3284-45ae-8809-96481a043957-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp\" (UID: \"cb487cf8-3284-45ae-8809-96481a043957\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" Nov 01 06:54:56.532236 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.532208 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5nlx\" (UniqueName: \"kubernetes.io/projected/cb487cf8-3284-45ae-8809-96481a043957-kube-api-access-d5nlx\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp\" (UID: \"cb487cf8-3284-45ae-8809-96481a043957\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" Nov 01 06:54:56.645727 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:56.645695 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" Nov 01 06:54:57.028931 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:57.028905 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp"] Nov 01 06:54:57.029652 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:54:57.029625 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb487cf8_3284_45ae_8809_96481a043957.slice/crio-9a107cbd7abee17bf8b1ce23d9a7f7623306a3f72c22db3007d1017681f430f4 WatchSource:0}: Error finding container 9a107cbd7abee17bf8b1ce23d9a7f7623306a3f72c22db3007d1017681f430f4: Status 404 returned error can't find the container with id 9a107cbd7abee17bf8b1ce23d9a7f7623306a3f72c22db3007d1017681f430f4 Nov 01 06:54:57.656820 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:57.656784 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" event={"ID":"cb487cf8-3284-45ae-8809-96481a043957","Type":"ContainerStarted","Data":"6dff2cc9624f58cdd7e5c4b1211be48630e9bbb7406e09a79f8fddb37fec1942"} Nov 01 06:54:57.657253 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:57.656853 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" event={"ID":"cb487cf8-3284-45ae-8809-96481a043957","Type":"ContainerStarted","Data":"9a107cbd7abee17bf8b1ce23d9a7f7623306a3f72c22db3007d1017681f430f4"} Nov 01 06:54:57.657253 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:57.656878 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" Nov 01 06:54:57.773382 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:57.773328 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" podStartSLOduration=1.7733139119999999 podStartE2EDuration="1.773313912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:54:57.772012708 +0000 UTC m=+2707.535812565" watchObservedRunningTime="2026-11-01 06:54:57.773313912 +0000 UTC m=+2707.537113768" Nov 01 06:54:58.437068 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:54:58.437036 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:03.121775 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:03.121745 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:06.859737 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:55:06.859699 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:55:08.140144 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:08.140102 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:08.667778 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:08.667751 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" Nov 01 06:55:11.991872 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:11.991816 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:12.135949 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:12.135913 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:12.774467 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:12.774426 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:17.170191 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:17.170152 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:17.414006 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:17.413963 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:17.976335 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:17.976288 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:18.121786 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:18.121752 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:22.153492 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:22.153455 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:24.738153 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:24.738117 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:25.669429 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:25.669394 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7686-3d97be"] Nov 01 06:55:25.673783 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:25.673759 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-3d97be" Nov 01 06:55:25.727440 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:25.727415 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-3d97be"] Nov 01 06:55:25.771975 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:25.771941 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-557zr\"" Nov 01 06:55:25.808918 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:25.808882 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5748t\" (UniqueName: \"kubernetes.io/projected/b600ea83-7582-4157-9527-a72f61b33c8c-kube-api-access-5748t\") pod \"test-curl-7686-3d97be\" (UID: \"b600ea83-7582-4157-9527-a72f61b33c8c\") " pod="opendatahub/test-curl-7686-3d97be" Nov 01 06:55:25.909781 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:25.909748 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5748t\" (UniqueName: \"kubernetes.io/projected/b600ea83-7582-4157-9527-a72f61b33c8c-kube-api-access-5748t\") pod \"test-curl-7686-3d97be\" (UID: \"b600ea83-7582-4157-9527-a72f61b33c8c\") " pod="opendatahub/test-curl-7686-3d97be" Nov 01 06:55:25.966119 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:25.966052 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5748t\" (UniqueName: \"kubernetes.io/projected/b600ea83-7582-4157-9527-a72f61b33c8c-kube-api-access-5748t\") pod \"test-curl-7686-3d97be\" (UID: \"b600ea83-7582-4157-9527-a72f61b33c8c\") " pod="opendatahub/test-curl-7686-3d97be" Nov 01 06:55:25.988008 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:25.987982 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-3d97be" Nov 01 06:55:26.361367 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:26.361339 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-3d97be"] Nov 01 06:55:26.361993 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:55:26.361969 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb600ea83_7582_4157_9527_a72f61b33c8c.slice/crio-e4bc75bca4790dac7c8e1b458317f1afa76bf94c0431b607ef4ba904e149d781 WatchSource:0}: Error finding container e4bc75bca4790dac7c8e1b458317f1afa76bf94c0431b607ef4ba904e149d781: Status 404 returned error can't find the container with id e4bc75bca4790dac7c8e1b458317f1afa76bf94c0431b607ef4ba904e149d781 Nov 01 06:55:26.916123 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:26.916087 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-3d97be" event={"ID":"b600ea83-7582-4157-9527-a72f61b33c8c","Type":"ContainerStarted","Data":"358cb8946cb780de3e8ed84b3cb71106423176a97e34890d9e5db510a18f98a7"} Nov 01 06:55:26.916524 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:26.916139 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-3d97be" event={"ID":"b600ea83-7582-4157-9527-a72f61b33c8c","Type":"ContainerStarted","Data":"e4bc75bca4790dac7c8e1b458317f1afa76bf94c0431b607ef4ba904e149d781"} Nov 01 06:55:27.029937 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:27.029877 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7686-3d97be" podStartSLOduration=2.029863824 podStartE2EDuration="2.029863824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:55:27.026823358 +0000 UTC m=+2736.790623217" watchObservedRunningTime="2026-11-01 06:55:27.029863824 +0000 UTC m=+2736.793663682" Nov 01 06:55:27.221306 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:27.221208 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:28.465725 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:28.465692 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7686-3d97be"] Nov 01 06:55:28.473514 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:28.473487 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7686-3d97be"] Nov 01 06:55:28.929901 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:28.929863 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7686-3d97be" podUID="b600ea83-7582-4157-9527-a72f61b33c8c" containerName="test-curl-7686-3d97be" containerID="cri-o://358cb8946cb780de3e8ed84b3cb71106423176a97e34890d9e5db510a18f98a7" gracePeriod=2 Nov 01 06:55:28.980771 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:55:28.980725 4385 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-3d97be\" is forbidden: User \"system:node:ip-10-0-140-68.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-68.us-west-1.compute.internal' and this object" podUID="b600ea83-7582-4157-9527-a72f61b33c8c" pod="opendatahub/test-curl-7686-3d97be" Nov 01 06:55:30.432400 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:55:30.432363 4385 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-3d97be\" is forbidden: User \"system:node:ip-10-0-140-68.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-68.us-west-1.compute.internal' and this object" podUID="b600ea83-7582-4157-9527-a72f61b33c8c" pod="opendatahub/test-curl-7686-3d97be" Nov 01 06:55:31.180805 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:31.180783 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-3d97be" Nov 01 06:55:31.368566 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:31.368529 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b600ea83-7582-4157-9527-a72f61b33c8c-kube-api-access-5748t\" (UniqueName: \"kubernetes.io/projected/b600ea83-7582-4157-9527-a72f61b33c8c-kube-api-access-5748t\") pod \"b600ea83-7582-4157-9527-a72f61b33c8c\" (UID: \"b600ea83-7582-4157-9527-a72f61b33c8c\") " Nov 01 06:55:31.370792 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:31.370769 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b600ea83-7582-4157-9527-a72f61b33c8c-kube-api-access-5748t" pod "b600ea83-7582-4157-9527-a72f61b33c8c" (UID: "b600ea83-7582-4157-9527-a72f61b33c8c"). InnerVolumeSpecName "kube-api-access-5748t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:55:31.470388 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:31.470352 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5748t\" (UniqueName: \"kubernetes.io/projected/b600ea83-7582-4157-9527-a72f61b33c8c-kube-api-access-5748t\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:55:31.921852 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:31.921808 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:31.952795 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:31.952760 4385 generic.go:356] "Generic (PLEG): container finished" podID="b600ea83-7582-4157-9527-a72f61b33c8c" containerID="358cb8946cb780de3e8ed84b3cb71106423176a97e34890d9e5db510a18f98a7" exitCode=137 Nov 01 06:55:31.952979 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:31.952800 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-3d97be" Nov 01 06:55:31.952979 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:31.952867 4385 scope.go:122] "RemoveContainer" containerID="358cb8946cb780de3e8ed84b3cb71106423176a97e34890d9e5db510a18f98a7" Nov 01 06:55:31.973370 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:31.973341 4385 scope.go:122] "RemoveContainer" containerID="358cb8946cb780de3e8ed84b3cb71106423176a97e34890d9e5db510a18f98a7" Nov 01 06:55:31.973779 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:55:31.973747 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"358cb8946cb780de3e8ed84b3cb71106423176a97e34890d9e5db510a18f98a7\": container with ID starting with 358cb8946cb780de3e8ed84b3cb71106423176a97e34890d9e5db510a18f98a7 not found: ID does not exist" containerID="358cb8946cb780de3e8ed84b3cb71106423176a97e34890d9e5db510a18f98a7" Nov 01 06:55:31.973971 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:31.973789 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"358cb8946cb780de3e8ed84b3cb71106423176a97e34890d9e5db510a18f98a7"} err="failed to get container status \"358cb8946cb780de3e8ed84b3cb71106423176a97e34890d9e5db510a18f98a7\": rpc error: code = NotFound desc = could not find container \"358cb8946cb780de3e8ed84b3cb71106423176a97e34890d9e5db510a18f98a7\": container with ID starting with 358cb8946cb780de3e8ed84b3cb71106423176a97e34890d9e5db510a18f98a7 not found: ID does not exist" Nov 01 06:55:32.016358 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:55:32.016319 4385 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-3d97be\" is forbidden: User \"system:node:ip-10-0-140-68.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-68.us-west-1.compute.internal' and this object" podUID="b600ea83-7582-4157-9527-a72f61b33c8c" pod="opendatahub/test-curl-7686-3d97be" Nov 01 06:55:32.385116 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:32.385086 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b600ea83-7582-4157-9527-a72f61b33c8c" path="/var/lib/kubelet/pods/b600ea83-7582-4157-9527-a72f61b33c8c/volumes" Nov 01 06:55:36.617106 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:36.617063 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:36.780026 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:55:36.779989 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:55:36.936971 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:36.936882 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:37.362565 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:37.362531 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:41.300821 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:41.300785 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:42.406795 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:42.406758 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:45.940412 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:45.940378 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:49.421294 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:49.421257 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:50.633681 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:50.633642 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:55:55.393371 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:55:55.393325 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:00.033284 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:00.033239 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:04.675811 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:04.675767 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:06.780940 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:56:06.780898 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:56:09.326036 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:09.325990 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:14.048263 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:14.048231 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:17.448001 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:17.447957 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp"] Nov 01 06:56:17.615197 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:17.615164 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:17.770032 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:17.769915 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:18.109702 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:18.109672 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:18.821625 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:18.821582 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:18.971637 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:18.971606 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:19.526489 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:19.526452 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:19.797612 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:19.797530 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:19.946806 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:19.946766 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:20.220555 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.220519 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng"] Nov 01 06:56:20.482663 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.482587 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:20.587978 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.587945 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x"] Nov 01 06:56:20.588578 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.588559 4385 cpu_manager.go:414] "containerMap: removing container" podUID="b600ea83-7582-4157-9527-a72f61b33c8c" containerName="test-curl-7686-3d97be" Nov 01 06:56:20.588686 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.588580 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b600ea83-7582-4157-9527-a72f61b33c8c" containerName="test-curl-7686-3d97be" Nov 01 06:56:20.588747 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.588719 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="b600ea83-7582-4157-9527-a72f61b33c8c" containerName="test-curl-7686-3d97be" Nov 01 06:56:20.592073 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.592050 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:20.631427 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.631389 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:20.644950 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.644923 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-302ba0\"/\"kube-root-ca.crt\"" Nov 01 06:56:20.645227 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.645201 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-302ba0\"/\"openshift-service-ca.crt\"" Nov 01 06:56:20.645374 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.645296 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-302ba0\"/\"sub-test-model-302ba0-kserve-self-signed-certs\"" Nov 01 06:56:20.650877 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.650816 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x"] Nov 01 06:56:20.653011 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.652990 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-home\") pod \"sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:20.653109 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.653036 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgf7v\" (UniqueName: \"kubernetes.io/projected/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-kube-api-access-wgf7v\") pod \"sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:20.653109 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.653064 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-model-cache\") pod \"sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:20.653232 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.653209 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-tmp-dir\") pod \"sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:20.653347 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.653264 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-tls-certs\") pod \"sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:20.653407 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.653383 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-dshm\") pod \"sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:20.694628 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.694595 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-302ba0\"/\"default-dockercfg-m8xdc\"" Nov 01 06:56:20.754460 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.754384 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-home\") pod \"sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:20.754460 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.754432 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgf7v\" (UniqueName: \"kubernetes.io/projected/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-kube-api-access-wgf7v\") pod \"sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:20.754654 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.754552 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-model-cache\") pod \"sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:20.754692 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.754650 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-tmp-dir\") pod \"sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:20.754731 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.754695 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-tls-certs\") pod \"sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:20.754768 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.754754 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-dshm\") pod \"sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:20.754888 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.754867 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-home\") pod \"sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:20.754954 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.754904 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-model-cache\") pod \"sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:20.755042 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.755021 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-tmp-dir\") pod \"sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:20.757244 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.757217 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-dshm\") pod \"sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:20.757542 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.757521 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-tls-certs\") pod \"sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:20.786036 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.786000 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:20.810881 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.810815 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgf7v\" (UniqueName: \"kubernetes.io/projected/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-kube-api-access-wgf7v\") pod \"sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:20.907760 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:20.907720 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:21.311677 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:21.311648 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x"] Nov 01 06:56:21.312436 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:56:21.312411 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f5b6430_dcf4_429c_bddc_f50bfba85c8f.slice/crio-bc3b22e2761ab67f4ef019f659af02780182f5b0416734b61ae167853b0f1549 WatchSource:0}: Error finding container bc3b22e2761ab67f4ef019f659af02780182f5b0416734b61ae167853b0f1549: Status 404 returned error can't find the container with id bc3b22e2761ab67f4ef019f659af02780182f5b0416734b61ae167853b0f1549 Nov 01 06:56:21.341317 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:21.341290 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" event={"ID":"4f5b6430-dcf4-429c-bddc-f50bfba85c8f","Type":"ContainerStarted","Data":"bc3b22e2761ab67f4ef019f659af02780182f5b0416734b61ae167853b0f1549"} Nov 01 06:56:22.350175 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:22.350138 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" event={"ID":"4f5b6430-dcf4-429c-bddc-f50bfba85c8f","Type":"ContainerStarted","Data":"d1eb3cf9f68d099b7fb9f63c39a3a885d18fd55ba906088ed28ffb0128295be5"} Nov 01 06:56:22.448324 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:22.448271 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" podUID="cb487cf8-3284-45ae-8809-96481a043957" containerName="payload-processing" containerID="cri-o://6dff2cc9624f58cdd7e5c4b1211be48630e9bbb7406e09a79f8fddb37fec1942" gracePeriod=25 Nov 01 06:56:22.466377 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:22.466326 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" podStartSLOduration=2.466311344 podStartE2EDuration="2.466311344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:56:22.463750536 +0000 UTC m=+2792.227550393" watchObservedRunningTime="2026-11-01 06:56:22.466311344 +0000 UTC m=+2792.230111200" Nov 01 06:56:22.729059 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:22.729030 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" Nov 01 06:56:22.882916 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:22.882877 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb487cf8-3284-45ae-8809-96481a043957-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb487cf8-3284-45ae-8809-96481a043957-provider-credentials\") pod \"cb487cf8-3284-45ae-8809-96481a043957\" (UID: \"cb487cf8-3284-45ae-8809-96481a043957\") " Nov 01 06:56:22.883100 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:22.883028 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb487cf8-3284-45ae-8809-96481a043957-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb487cf8-3284-45ae-8809-96481a043957-plugins-config-volume\") pod \"cb487cf8-3284-45ae-8809-96481a043957\" (UID: \"cb487cf8-3284-45ae-8809-96481a043957\") " Nov 01 06:56:22.883174 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:22.883117 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb487cf8-3284-45ae-8809-96481a043957-kube-api-access-d5nlx\" (UniqueName: \"kubernetes.io/projected/cb487cf8-3284-45ae-8809-96481a043957-kube-api-access-d5nlx\") pod \"cb487cf8-3284-45ae-8809-96481a043957\" (UID: \"cb487cf8-3284-45ae-8809-96481a043957\") " Nov 01 06:56:22.883231 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:22.883181 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb487cf8-3284-45ae-8809-96481a043957-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb487cf8-3284-45ae-8809-96481a043957-routing-overlay-volume\") pod \"cb487cf8-3284-45ae-8809-96481a043957\" (UID: \"cb487cf8-3284-45ae-8809-96481a043957\") " Nov 01 06:56:22.883511 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:22.883483 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb487cf8-3284-45ae-8809-96481a043957-plugins-config-volume" pod "cb487cf8-3284-45ae-8809-96481a043957" (UID: "cb487cf8-3284-45ae-8809-96481a043957"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:56:22.883769 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:22.883728 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb487cf8-3284-45ae-8809-96481a043957-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:56:22.883769 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:22.883740 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb487cf8-3284-45ae-8809-96481a043957-routing-overlay-volume" pod "cb487cf8-3284-45ae-8809-96481a043957" (UID: "cb487cf8-3284-45ae-8809-96481a043957"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:56:22.885163 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:22.885141 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb487cf8-3284-45ae-8809-96481a043957-provider-credentials" pod "cb487cf8-3284-45ae-8809-96481a043957" (UID: "cb487cf8-3284-45ae-8809-96481a043957"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:56:22.885543 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:22.885510 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb487cf8-3284-45ae-8809-96481a043957-kube-api-access-d5nlx" pod "cb487cf8-3284-45ae-8809-96481a043957" (UID: "cb487cf8-3284-45ae-8809-96481a043957"). InnerVolumeSpecName "kube-api-access-d5nlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:56:22.908335 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:22.908312 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:22.985135 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:22.985101 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5nlx\" (UniqueName: \"kubernetes.io/projected/cb487cf8-3284-45ae-8809-96481a043957-kube-api-access-d5nlx\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:56:22.985135 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:22.985130 4385 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb487cf8-3284-45ae-8809-96481a043957-routing-overlay-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:56:22.985135 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:22.985139 4385 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb487cf8-3284-45ae-8809-96481a043957-provider-credentials\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:56:23.358658 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:23.358628 4385 generic.go:356] "Generic (PLEG): container finished" podID="cb487cf8-3284-45ae-8809-96481a043957" containerID="6dff2cc9624f58cdd7e5c4b1211be48630e9bbb7406e09a79f8fddb37fec1942" exitCode=0 Nov 01 06:56:23.359123 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:23.358678 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" Nov 01 06:56:23.359123 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:23.358716 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" event={"ID":"cb487cf8-3284-45ae-8809-96481a043957","Type":"ContainerDied","Data":"6dff2cc9624f58cdd7e5c4b1211be48630e9bbb7406e09a79f8fddb37fec1942"} Nov 01 06:56:23.359123 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:23.358772 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp" event={"ID":"cb487cf8-3284-45ae-8809-96481a043957","Type":"ContainerDied","Data":"9a107cbd7abee17bf8b1ce23d9a7f7623306a3f72c22db3007d1017681f430f4"} Nov 01 06:56:23.359123 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:23.358799 4385 scope.go:122] "RemoveContainer" containerID="6dff2cc9624f58cdd7e5c4b1211be48630e9bbb7406e09a79f8fddb37fec1942" Nov 01 06:56:23.370752 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:23.370726 4385 scope.go:122] "RemoveContainer" containerID="6dff2cc9624f58cdd7e5c4b1211be48630e9bbb7406e09a79f8fddb37fec1942" Nov 01 06:56:23.371131 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:56:23.371110 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dff2cc9624f58cdd7e5c4b1211be48630e9bbb7406e09a79f8fddb37fec1942\": container with ID starting with 6dff2cc9624f58cdd7e5c4b1211be48630e9bbb7406e09a79f8fddb37fec1942 not found: ID does not exist" containerID="6dff2cc9624f58cdd7e5c4b1211be48630e9bbb7406e09a79f8fddb37fec1942" Nov 01 06:56:23.371219 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:23.371138 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dff2cc9624f58cdd7e5c4b1211be48630e9bbb7406e09a79f8fddb37fec1942"} err="failed to get container status \"6dff2cc9624f58cdd7e5c4b1211be48630e9bbb7406e09a79f8fddb37fec1942\": rpc error: code = NotFound desc = could not find container \"6dff2cc9624f58cdd7e5c4b1211be48630e9bbb7406e09a79f8fddb37fec1942\": container with ID starting with 6dff2cc9624f58cdd7e5c4b1211be48630e9bbb7406e09a79f8fddb37fec1942 not found: ID does not exist" Nov 01 06:56:23.523093 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:23.523056 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp"] Nov 01 06:56:23.574227 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:23.574196 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-65fb27shp"] Nov 01 06:56:23.827809 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:23.827776 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:24.278210 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:24.278110 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:24.388504 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:24.388471 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb487cf8-3284-45ae-8809-96481a043957" path="/var/lib/kubelet/pods/cb487cf8-3284-45ae-8809-96481a043957/volumes" Nov 01 06:56:24.445652 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:24.445615 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:24.608119 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:24.608070 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:25.222007 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:25.221955 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng" podUID="15b0a39a-6939-433a-803a-8cdaf6dac909" containerName="payload-pre-processing" containerID="cri-o://b7c9ca05def068f3a29b90c63ad78dc5a14e6532330b56744236ae7dc59d81b0" gracePeriod=25 Nov 01 06:56:25.376141 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:25.376098 4385 generic.go:356] "Generic (PLEG): container finished" podID="15b0a39a-6939-433a-803a-8cdaf6dac909" containerID="b7c9ca05def068f3a29b90c63ad78dc5a14e6532330b56744236ae7dc59d81b0" exitCode=0 Nov 01 06:56:25.376402 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:25.376381 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng" event={"ID":"15b0a39a-6939-433a-803a-8cdaf6dac909","Type":"ContainerDied","Data":"b7c9ca05def068f3a29b90c63ad78dc5a14e6532330b56744236ae7dc59d81b0"} Nov 01 06:56:25.422771 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:25.422734 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:25.497020 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:25.496994 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng" Nov 01 06:56:25.618004 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:25.617971 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15b0a39a-6939-433a-803a-8cdaf6dac909-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15b0a39a-6939-433a-803a-8cdaf6dac909-plugins-config-volume\") pod \"15b0a39a-6939-433a-803a-8cdaf6dac909\" (UID: \"15b0a39a-6939-433a-803a-8cdaf6dac909\") " Nov 01 06:56:25.618190 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:25.618170 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b0a39a-6939-433a-803a-8cdaf6dac909-kube-api-access-lt4fg\" (UniqueName: \"kubernetes.io/projected/15b0a39a-6939-433a-803a-8cdaf6dac909-kube-api-access-lt4fg\") pod \"15b0a39a-6939-433a-803a-8cdaf6dac909\" (UID: \"15b0a39a-6939-433a-803a-8cdaf6dac909\") " Nov 01 06:56:25.620292 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:25.620241 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b0a39a-6939-433a-803a-8cdaf6dac909-plugins-config-volume" pod "15b0a39a-6939-433a-803a-8cdaf6dac909" (UID: "15b0a39a-6939-433a-803a-8cdaf6dac909"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:56:25.624176 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:25.624146 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b0a39a-6939-433a-803a-8cdaf6dac909-kube-api-access-lt4fg" pod "15b0a39a-6939-433a-803a-8cdaf6dac909" (UID: "15b0a39a-6939-433a-803a-8cdaf6dac909"). InnerVolumeSpecName "kube-api-access-lt4fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:56:25.719308 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:25.719274 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt4fg\" (UniqueName: \"kubernetes.io/projected/15b0a39a-6939-433a-803a-8cdaf6dac909-kube-api-access-lt4fg\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:56:25.719308 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:25.719301 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15b0a39a-6939-433a-803a-8cdaf6dac909-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:56:26.386054 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:26.386024 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng" Nov 01 06:56:26.386054 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:26.386048 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng" event={"ID":"15b0a39a-6939-433a-803a-8cdaf6dac909","Type":"ContainerDied","Data":"8edc56e3f42f8a4efa46e2d76df2aa11f47e2727ac090713ae1324ea43da71b5"} Nov 01 06:56:26.386378 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:26.386131 4385 scope.go:122] "RemoveContainer" containerID="b7c9ca05def068f3a29b90c63ad78dc5a14e6532330b56744236ae7dc59d81b0" Nov 01 06:56:26.555639 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:26.555605 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng"] Nov 01 06:56:26.603928 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:26.603864 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a3e841-6d597c6575-nn9ng"] Nov 01 06:56:28.385958 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:28.385928 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15b0a39a-6939-433a-803a-8cdaf6dac909" path="/var/lib/kubelet/pods/15b0a39a-6939-433a-803a-8cdaf6dac909/volumes" Nov 01 06:56:28.534992 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:28.534957 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:28.704379 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:28.704271 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:28.945692 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:28.945654 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:29.079277 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:29.078010 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn"] Nov 01 06:56:29.079277 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:29.078818 4385 cpu_manager.go:414] "containerMap: removing container" podUID="cb487cf8-3284-45ae-8809-96481a043957" containerName="payload-processing" Nov 01 06:56:29.079277 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:29.078859 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb487cf8-3284-45ae-8809-96481a043957" containerName="payload-processing" Nov 01 06:56:29.079277 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:29.078925 4385 cpu_manager.go:414] "containerMap: removing container" podUID="15b0a39a-6939-433a-803a-8cdaf6dac909" containerName="payload-pre-processing" Nov 01 06:56:29.079277 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:29.078933 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b0a39a-6939-433a-803a-8cdaf6dac909" containerName="payload-pre-processing" Nov 01 06:56:29.079277 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:29.079065 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b0a39a-6939-433a-803a-8cdaf6dac909" containerName="payload-pre-processing" Nov 01 06:56:29.079277 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:29.079080 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb487cf8-3284-45ae-8809-96481a043957" containerName="payload-processing" Nov 01 06:56:29.084647 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:29.084621 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn" Nov 01 06:56:29.134774 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:29.134741 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn"] Nov 01 06:56:29.137105 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:29.137078 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-3db672\"" Nov 01 06:56:29.174029 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:29.173983 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvjv4\" (UniqueName: \"kubernetes.io/projected/7d064de5-6ab4-45ab-a83a-0fe33a874788-kube-api-access-jvjv4\") pod \"payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn\" (UID: \"7d064de5-6ab4-45ab-a83a-0fe33a874788\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn" Nov 01 06:56:29.174228 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:29.174044 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d064de5-6ab4-45ab-a83a-0fe33a874788-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn\" (UID: \"7d064de5-6ab4-45ab-a83a-0fe33a874788\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn" Nov 01 06:56:29.180121 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:29.180100 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-3db672-dockercfg-85lhg\"" Nov 01 06:56:29.274705 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:29.274666 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvjv4\" (UniqueName: \"kubernetes.io/projected/7d064de5-6ab4-45ab-a83a-0fe33a874788-kube-api-access-jvjv4\") pod \"payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn\" (UID: \"7d064de5-6ab4-45ab-a83a-0fe33a874788\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn" Nov 01 06:56:29.274895 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:29.274737 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d064de5-6ab4-45ab-a83a-0fe33a874788-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn\" (UID: \"7d064de5-6ab4-45ab-a83a-0fe33a874788\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn" Nov 01 06:56:29.275602 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:29.275578 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d064de5-6ab4-45ab-a83a-0fe33a874788-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn\" (UID: \"7d064de5-6ab4-45ab-a83a-0fe33a874788\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn" Nov 01 06:56:29.331311 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:29.331244 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvjv4\" (UniqueName: \"kubernetes.io/projected/7d064de5-6ab4-45ab-a83a-0fe33a874788-kube-api-access-jvjv4\") pod \"payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn\" (UID: \"7d064de5-6ab4-45ab-a83a-0fe33a874788\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn" Nov 01 06:56:29.404326 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:29.404288 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn" Nov 01 06:56:29.786932 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:29.786903 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn"] Nov 01 06:56:29.787083 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:56:29.787027 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d064de5_6ab4_45ab_a83a_0fe33a874788.slice/crio-faafcbfdac2b37995db72f5466408661672b618e7cd58f5f9d88dd722cb23acd WatchSource:0}: Error finding container faafcbfdac2b37995db72f5466408661672b618e7cd58f5f9d88dd722cb23acd: Status 404 returned error can't find the container with id faafcbfdac2b37995db72f5466408661672b618e7cd58f5f9d88dd722cb23acd Nov 01 06:56:30.419371 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:30.419330 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn" event={"ID":"7d064de5-6ab4-45ab-a83a-0fe33a874788","Type":"ContainerStarted","Data":"567fd33c3a411e072e2e0fa93bd813557c820661d7f7d0771f5c713eb658e74d"} Nov 01 06:56:30.419773 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:30.419403 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn" event={"ID":"7d064de5-6ab4-45ab-a83a-0fe33a874788","Type":"ContainerStarted","Data":"faafcbfdac2b37995db72f5466408661672b618e7cd58f5f9d88dd722cb23acd"} Nov 01 06:56:30.419773 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:30.419542 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn" Nov 01 06:56:30.533392 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:30.533342 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn" podStartSLOduration=1.533326492 podStartE2EDuration="1.533326492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:56:30.531748927 +0000 UTC m=+2800.295548785" watchObservedRunningTime="2026-11-01 06:56:30.533326492 +0000 UTC m=+2800.297126350" Nov 01 06:56:30.908552 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:30.908498 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:30.921982 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:30.921957 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:31.438059 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:31.438031 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 06:56:33.599574 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:33.599527 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:36.781190 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:56:36.781152 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:56:38.226235 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:38.226202 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:40.600380 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:40.600346 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:41.224287 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:41.224221 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:41.430516 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:41.430490 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn" Nov 01 06:56:41.600808 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:41.600551 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:42.296400 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:42.296364 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58"] Nov 01 06:56:42.296607 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:42.296584 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" podUID="7efc28e3-9824-4c6b-ae7e-e9323ee6a130" containerName="maas-api" containerID="cri-o://d4e34b3167a3166dab408adefa0ae5bc6d5a1f899a2f411d6e81f27260e9cf0d" gracePeriod=30 Nov 01 06:56:42.422491 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:42.422460 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:42.517773 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:42.517739 4385 generic.go:356] "Generic (PLEG): container finished" podID="7efc28e3-9824-4c6b-ae7e-e9323ee6a130" containerID="d4e34b3167a3166dab408adefa0ae5bc6d5a1f899a2f411d6e81f27260e9cf0d" exitCode=0 Nov 01 06:56:42.518222 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:42.518193 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" event={"ID":"7efc28e3-9824-4c6b-ae7e-e9323ee6a130","Type":"ContainerDied","Data":"d4e34b3167a3166dab408adefa0ae5bc6d5a1f899a2f411d6e81f27260e9cf0d"} Nov 01 06:56:42.589142 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:42.589115 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" Nov 01 06:56:42.736376 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:42.736325 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-kube-api-access-b869d\" (UniqueName: \"kubernetes.io/projected/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-kube-api-access-b869d\") pod \"7efc28e3-9824-4c6b-ae7e-e9323ee6a130\" (UID: \"7efc28e3-9824-4c6b-ae7e-e9323ee6a130\") " Nov 01 06:56:42.736909 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:42.736462 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-maas-api-tls\") pod \"7efc28e3-9824-4c6b-ae7e-e9323ee6a130\" (UID: \"7efc28e3-9824-4c6b-ae7e-e9323ee6a130\") " Nov 01 06:56:42.736909 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:42.736493 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-metrics-certs\") pod \"7efc28e3-9824-4c6b-ae7e-e9323ee6a130\" (UID: \"7efc28e3-9824-4c6b-ae7e-e9323ee6a130\") " Nov 01 06:56:42.736909 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:42.736545 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-openshift-service-ca\") pod \"7efc28e3-9824-4c6b-ae7e-e9323ee6a130\" (UID: \"7efc28e3-9824-4c6b-ae7e-e9323ee6a130\") " Nov 01 06:56:42.737463 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:42.737437 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-openshift-service-ca" pod "7efc28e3-9824-4c6b-ae7e-e9323ee6a130" (UID: "7efc28e3-9824-4c6b-ae7e-e9323ee6a130"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:56:42.740115 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:42.740073 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-maas-api-tls" pod "7efc28e3-9824-4c6b-ae7e-e9323ee6a130" (UID: "7efc28e3-9824-4c6b-ae7e-e9323ee6a130"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:56:42.740214 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:42.740115 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-metrics-certs" pod "7efc28e3-9824-4c6b-ae7e-e9323ee6a130" (UID: "7efc28e3-9824-4c6b-ae7e-e9323ee6a130"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:56:42.740214 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:42.740134 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-kube-api-access-b869d" pod "7efc28e3-9824-4c6b-ae7e-e9323ee6a130" (UID: "7efc28e3-9824-4c6b-ae7e-e9323ee6a130"). InnerVolumeSpecName "kube-api-access-b869d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:56:42.838266 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:42.838220 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-maas-api-tls\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:56:42.838266 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:42.838259 4385 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-metrics-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:56:42.838545 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:42.838277 4385 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-openshift-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:56:42.838545 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:42.838293 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b869d\" (UniqueName: \"kubernetes.io/projected/7efc28e3-9824-4c6b-ae7e-e9323ee6a130-kube-api-access-b869d\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:56:43.526247 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:43.526219 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" Nov 01 06:56:43.526446 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:43.526244 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58" event={"ID":"7efc28e3-9824-4c6b-ae7e-e9323ee6a130","Type":"ContainerDied","Data":"6647b900d3394b42aad3fa2e5b7b83cab070a89c1d28b2faabea8cceb4f58304"} Nov 01 06:56:43.526446 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:43.526305 4385 scope.go:122] "RemoveContainer" containerID="d4e34b3167a3166dab408adefa0ae5bc6d5a1f899a2f411d6e81f27260e9cf0d" Nov 01 06:56:43.690805 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:43.690690 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58"] Nov 01 06:56:43.740579 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:43.740553 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a3e841-849cdc6685-9wc58"] Nov 01 06:56:44.267408 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:44.267375 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:44.385466 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:44.385433 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7efc28e3-9824-4c6b-ae7e-e9323ee6a130" path="/var/lib/kubelet/pods/7efc28e3-9824-4c6b-ae7e-e9323ee6a130/volumes" Nov 01 06:56:45.330128 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:45.330094 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:45.508147 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:45.508108 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:45.665574 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:45.665477 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:48.272848 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.272803 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:48.418715 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.418685 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:48.680882 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.680849 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6"] Nov 01 06:56:48.681409 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.681395 4385 cpu_manager.go:414] "containerMap: removing container" podUID="7efc28e3-9824-4c6b-ae7e-e9323ee6a130" containerName="maas-api" Nov 01 06:56:48.681479 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.681410 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7efc28e3-9824-4c6b-ae7e-e9323ee6a130" containerName="maas-api" Nov 01 06:56:48.681532 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.681502 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="7efc28e3-9824-4c6b-ae7e-e9323ee6a130" containerName="maas-api" Nov 01 06:56:48.684818 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.684799 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:48.756793 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.756763 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-6ba6e5\"/\"openshift-service-ca.crt\"" Nov 01 06:56:48.762677 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.762641 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6"] Nov 01 06:56:48.778525 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.778495 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:48.787430 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.787403 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-6ba6e5\"/\"kube-root-ca.crt\"" Nov 01 06:56:48.787569 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.787505 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-6ba6e5\"/\"sub-test-model-6ba6e5-kserve-self-signed-certs\"" Nov 01 06:56:48.787644 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.787583 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-6ba6e5\"/\"default-dockercfg-c6zdt\"" Nov 01 06:56:48.813172 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.813140 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67193eb4-e54e-4b57-b9f1-6cf576af70f7-tls-certs\") pod \"sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:48.813339 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.813178 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-model-cache\") pod \"sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:48.813339 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.813260 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnxl6\" (UniqueName: \"kubernetes.io/projected/67193eb4-e54e-4b57-b9f1-6cf576af70f7-kube-api-access-bnxl6\") pod \"sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:48.813339 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.813287 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-home\") pod \"sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:48.813339 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.813308 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-dshm\") pod \"sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:48.813494 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.813477 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-tmp-dir\") pod \"sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:48.914683 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.914638 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67193eb4-e54e-4b57-b9f1-6cf576af70f7-tls-certs\") pod \"sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:48.914892 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.914697 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-model-cache\") pod \"sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:48.914892 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.914746 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnxl6\" (UniqueName: \"kubernetes.io/projected/67193eb4-e54e-4b57-b9f1-6cf576af70f7-kube-api-access-bnxl6\") pod \"sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:48.914892 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.914777 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-home\") pod \"sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:48.914892 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.914801 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-dshm\") pod \"sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:48.915128 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.914915 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-tmp-dir\") pod \"sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:48.915275 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.915247 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-model-cache\") pod \"sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:48.915365 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.915340 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-tmp-dir\") pod \"sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:48.915528 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.915509 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-home\") pod \"sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:48.917574 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.917546 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-dshm\") pod \"sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:48.918155 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.918135 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67193eb4-e54e-4b57-b9f1-6cf576af70f7-tls-certs\") pod \"sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:48.928376 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.928347 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:48.974817 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.974741 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnxl6\" (UniqueName: \"kubernetes.io/projected/67193eb4-e54e-4b57-b9f1-6cf576af70f7-kube-api-access-bnxl6\") pod \"sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:48.999518 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:48.999475 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:49.100415 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:49.099821 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:49.426858 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:49.416544 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6"] Nov 01 06:56:49.498873 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:49.498802 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n"] Nov 01 06:56:49.504245 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:49.504213 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n" Nov 01 06:56:49.533810 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:49.533785 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n"] Nov 01 06:56:49.557655 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:49.556962 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c831a4c0\"" Nov 01 06:56:49.557655 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:49.557498 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c831a4c0-dockercfg-6pk2x\"" Nov 01 06:56:49.582594 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:49.582557 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:49.583132 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:49.583100 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" event={"ID":"67193eb4-e54e-4b57-b9f1-6cf576af70f7","Type":"ContainerStarted","Data":"7535425b5544cfc5f35c2982da2471f7b493e44e8e4d5c3f63b5a4ac9e8fd7a2"} Nov 01 06:56:49.643495 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:49.643456 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgxxn\" (UniqueName: \"kubernetes.io/projected/250ce347-5c48-4490-b942-0f4916c707a5-kube-api-access-cgxxn\") pod \"payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n\" (UID: \"250ce347-5c48-4490-b942-0f4916c707a5\") " pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n" Nov 01 06:56:49.643935 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:49.643906 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250ce347-5c48-4490-b942-0f4916c707a5-plugins-config-volume\") pod \"payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n\" (UID: \"250ce347-5c48-4490-b942-0f4916c707a5\") " pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n" Nov 01 06:56:49.744897 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:49.744779 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgxxn\" (UniqueName: \"kubernetes.io/projected/250ce347-5c48-4490-b942-0f4916c707a5-kube-api-access-cgxxn\") pod \"payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n\" (UID: \"250ce347-5c48-4490-b942-0f4916c707a5\") " pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n" Nov 01 06:56:49.745086 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:49.744935 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250ce347-5c48-4490-b942-0f4916c707a5-plugins-config-volume\") pod \"payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n\" (UID: \"250ce347-5c48-4490-b942-0f4916c707a5\") " pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n" Nov 01 06:56:49.745589 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:49.745568 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250ce347-5c48-4490-b942-0f4916c707a5-plugins-config-volume\") pod \"payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n\" (UID: \"250ce347-5c48-4490-b942-0f4916c707a5\") " pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n" Nov 01 06:56:49.768531 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:49.768496 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:49.801562 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:49.801525 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgxxn\" (UniqueName: \"kubernetes.io/projected/250ce347-5c48-4490-b942-0f4916c707a5-kube-api-access-cgxxn\") pod \"payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n\" (UID: \"250ce347-5c48-4490-b942-0f4916c707a5\") " pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n" Nov 01 06:56:49.826035 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:49.825999 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n" Nov 01 06:56:49.913270 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:49.913227 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:50.212775 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:56:50.212740 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250ce347_5c48_4490_b942_0f4916c707a5.slice/crio-4361fef4ee46f6c4def71186087d51100fcb510ae50971b03b1c3ce513449f41 WatchSource:0}: Error finding container 4361fef4ee46f6c4def71186087d51100fcb510ae50971b03b1c3ce513449f41: Status 404 returned error can't find the container with id 4361fef4ee46f6c4def71186087d51100fcb510ae50971b03b1c3ce513449f41 Nov 01 06:56:50.212959 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:50.212796 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n"] Nov 01 06:56:50.343099 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:50.343066 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:50.605705 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:50.605666 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n" event={"ID":"250ce347-5c48-4490-b942-0f4916c707a5","Type":"ContainerStarted","Data":"4b481459890a78fc9ffa780bacef97b5d90981e85d00f01696fc67ab1aabce67"} Nov 01 06:56:50.606335 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:50.606313 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n" event={"ID":"250ce347-5c48-4490-b942-0f4916c707a5","Type":"ContainerStarted","Data":"4361fef4ee46f6c4def71186087d51100fcb510ae50971b03b1c3ce513449f41"} Nov 01 06:56:50.606501 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:50.606481 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n" Nov 01 06:56:50.607658 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:50.607631 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" event={"ID":"67193eb4-e54e-4b57-b9f1-6cf576af70f7","Type":"ContainerStarted","Data":"375ea6b05e18f40ee83eb05b75b219f0e5da780bab11de27f66f3e5492ae734c"} Nov 01 06:56:50.721411 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:50.721346 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n" podStartSLOduration=1.721331299 podStartE2EDuration="1.721331299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:56:50.719069532 +0000 UTC m=+2820.482869392" watchObservedRunningTime="2026-11-01 06:56:50.721331299 +0000 UTC m=+2820.485131156" Nov 01 06:56:50.838705 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:50.838648 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" podStartSLOduration=2.838633641 podStartE2EDuration="2.838633641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:56:50.836653776 +0000 UTC m=+2820.600453633" watchObservedRunningTime="2026-11-01 06:56:50.838633641 +0000 UTC m=+2820.602433532" Nov 01 06:56:50.941540 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:50.941450 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:50.999983 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:50.999938 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:51.590856 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:51.588609 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:51.806573 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:51.806539 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:54.699494 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:54.699460 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:55.512991 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:55.512952 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:55.681687 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:55.681642 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:55.829222 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:55.829188 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:58.697052 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:58.697005 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:58.888279 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:58.888251 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:56:59.011918 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:59.003697 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:59.023072 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:59.023042 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:59.129388 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:59.129354 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj"] Nov 01 06:56:59.134677 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:59.134656 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj" Nov 01 06:56:59.145998 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:59.145975 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj"] Nov 01 06:56:59.170739 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:59.170696 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8vvb\" (UniqueName: \"kubernetes.io/projected/22d2e118-2604-4c39-9f1a-cf6a509397fa-kube-api-access-c8vvb\") pod \"payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj\" (UID: \"22d2e118-2604-4c39-9f1a-cf6a509397fa\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj" Nov 01 06:56:59.171002 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:59.170798 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d2e118-2604-4c39-9f1a-cf6a509397fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj\" (UID: \"22d2e118-2604-4c39-9f1a-cf6a509397fa\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj" Nov 01 06:56:59.187802 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:59.187770 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-0cb4ef\"" Nov 01 06:56:59.196986 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:59.196156 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-0cb4ef-dockercfg-qxjg2\"" Nov 01 06:56:59.272124 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:59.272035 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d2e118-2604-4c39-9f1a-cf6a509397fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj\" (UID: \"22d2e118-2604-4c39-9f1a-cf6a509397fa\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj" Nov 01 06:56:59.272303 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:59.272247 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8vvb\" (UniqueName: \"kubernetes.io/projected/22d2e118-2604-4c39-9f1a-cf6a509397fa-kube-api-access-c8vvb\") pod \"payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj\" (UID: \"22d2e118-2604-4c39-9f1a-cf6a509397fa\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj" Nov 01 06:56:59.273004 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:59.272979 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d2e118-2604-4c39-9f1a-cf6a509397fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj\" (UID: \"22d2e118-2604-4c39-9f1a-cf6a509397fa\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj" Nov 01 06:56:59.328473 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:59.328446 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8vvb\" (UniqueName: \"kubernetes.io/projected/22d2e118-2604-4c39-9f1a-cf6a509397fa-kube-api-access-c8vvb\") pod \"payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj\" (UID: \"22d2e118-2604-4c39-9f1a-cf6a509397fa\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj" Nov 01 06:56:59.450462 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:59.450429 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj" Nov 01 06:56:59.751704 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:59.751679 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 06:56:59.830644 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:56:59.830605 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj"] Nov 01 06:56:59.831093 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:56:59.831068 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d2e118_2604_4c39_9f1a_cf6a509397fa.slice/crio-c41b0623271572ac6db1ec42b6d833fa6f0a8d4581fb6d677bfc97c7395f4ee1 WatchSource:0}: Error finding container c41b0623271572ac6db1ec42b6d833fa6f0a8d4581fb6d677bfc97c7395f4ee1: Status 404 returned error can't find the container with id c41b0623271572ac6db1ec42b6d833fa6f0a8d4581fb6d677bfc97c7395f4ee1 Nov 01 06:57:00.040640 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:00.040544 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:57:00.749961 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:00.749929 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj" event={"ID":"22d2e118-2604-4c39-9f1a-cf6a509397fa","Type":"ContainerStarted","Data":"d37fb46e317d9677017fbad1ea6eabbba0e8b7af5f017e8fbea33e4ea9456e9c"} Nov 01 06:57:00.750160 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:00.749992 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj" event={"ID":"22d2e118-2604-4c39-9f1a-cf6a509397fa","Type":"ContainerStarted","Data":"c41b0623271572ac6db1ec42b6d833fa6f0a8d4581fb6d677bfc97c7395f4ee1"} Nov 01 06:57:00.750160 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:00.750051 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj" Nov 01 06:57:00.862543 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:00.862493 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj" podStartSLOduration=1.862480189 podStartE2EDuration="1.862480189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:57:00.861393165 +0000 UTC m=+2830.625193022" watchObservedRunningTime="2026-11-01 06:57:00.862480189 +0000 UTC m=+2830.626280045" Nov 01 06:57:02.698802 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:02.698723 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n" Nov 01 06:57:04.727606 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:04.727568 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:57:06.781571 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:57:06.781531 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:57:09.437896 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:09.437855 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:57:11.761359 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:11.761333 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj" Nov 01 06:57:14.116348 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:14.116302 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:57:18.785140 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:18.785108 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:57:23.421253 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:23.421215 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:57:27.449388 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:27.449357 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:57:28.061070 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:28.061031 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:57:32.754261 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:32.754224 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:57:36.780476 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:57:36.780436 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:57:37.305223 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:37.305184 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:57:37.461308 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:37.461269 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:57:37.602074 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:37.602033 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:57:42.128217 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:42.128162 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:57:46.913060 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:46.913007 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:57:50.106665 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:50.106628 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:57:51.619161 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:51.619119 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:57:54.169032 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:54.169000 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:57:56.199292 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:56.199254 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:57:56.402126 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:57:56.402089 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:58:00.664693 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:00.664664 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6"] Nov 01 06:58:00.668624 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:00.668599 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" Nov 01 06:58:00.720562 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:00.720531 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c831a4c0-serving-cert\"" Nov 01 06:58:00.774241 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:00.774204 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6"] Nov 01 06:58:00.807825 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:00.807779 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgp28\" (UniqueName: \"kubernetes.io/projected/c42d21f9-d165-4278-b571-678356d3906e-kube-api-access-sgp28\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6\" (UID: \"c42d21f9-d165-4278-b571-678356d3906e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" Nov 01 06:58:00.808223 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:00.808202 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42d21f9-d165-4278-b571-678356d3906e-openshift-service-ca\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6\" (UID: \"c42d21f9-d165-4278-b571-678356d3906e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" Nov 01 06:58:00.808431 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:00.808404 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c42d21f9-d165-4278-b571-678356d3906e-maas-api-tls\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6\" (UID: \"c42d21f9-d165-4278-b571-678356d3906e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" Nov 01 06:58:00.808692 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:00.808674 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c42d21f9-d165-4278-b571-678356d3906e-metrics-certs\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6\" (UID: \"c42d21f9-d165-4278-b571-678356d3906e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" Nov 01 06:58:00.859677 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:00.859643 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:58:00.910011 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:00.909712 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c42d21f9-d165-4278-b571-678356d3906e-maas-api-tls\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6\" (UID: \"c42d21f9-d165-4278-b571-678356d3906e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" Nov 01 06:58:00.910011 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:00.909859 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c42d21f9-d165-4278-b571-678356d3906e-metrics-certs\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6\" (UID: \"c42d21f9-d165-4278-b571-678356d3906e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" Nov 01 06:58:00.910011 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:58:00.909886 4385 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-serving-cert: secret "maas-api-e2e-mt-c831a4c0-serving-cert" not found Nov 01 06:58:00.910011 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:58:00.910014 4385 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42d21f9-d165-4278-b571-678356d3906e-maas-api-tls podName:c42d21f9-d165-4278-b571-678356d3906e nodeName:}" failed. No retries permitted until 2026-11-01 06:58:01.409993273 +0000 UTC m=+2891.173793124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c42d21f9-d165-4278-b571-678356d3906e-maas-api-tls") pod "maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" (UID: "c42d21f9-d165-4278-b571-678356d3906e") : secret "maas-api-e2e-mt-c831a4c0-serving-cert" not found Nov 01 06:58:00.910428 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:00.910099 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgp28\" (UniqueName: \"kubernetes.io/projected/c42d21f9-d165-4278-b571-678356d3906e-kube-api-access-sgp28\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6\" (UID: \"c42d21f9-d165-4278-b571-678356d3906e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" Nov 01 06:58:00.910428 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:00.910171 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42d21f9-d165-4278-b571-678356d3906e-openshift-service-ca\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6\" (UID: \"c42d21f9-d165-4278-b571-678356d3906e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" Nov 01 06:58:00.911070 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:00.911045 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42d21f9-d165-4278-b571-678356d3906e-openshift-service-ca\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6\" (UID: \"c42d21f9-d165-4278-b571-678356d3906e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" Nov 01 06:58:00.918910 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:00.913348 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c42d21f9-d165-4278-b571-678356d3906e-metrics-certs\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6\" (UID: \"c42d21f9-d165-4278-b571-678356d3906e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" Nov 01 06:58:00.968790 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:00.968763 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgp28\" (UniqueName: \"kubernetes.io/projected/c42d21f9-d165-4278-b571-678356d3906e-kube-api-access-sgp28\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6\" (UID: \"c42d21f9-d165-4278-b571-678356d3906e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" Nov 01 06:58:01.019309 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:01.019269 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:58:01.164329 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:01.164287 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:58:01.421470 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:01.421440 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c42d21f9-d165-4278-b571-678356d3906e-maas-api-tls\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6\" (UID: \"c42d21f9-d165-4278-b571-678356d3906e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" Nov 01 06:58:01.424178 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:01.424150 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c42d21f9-d165-4278-b571-678356d3906e-maas-api-tls\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6\" (UID: \"c42d21f9-d165-4278-b571-678356d3906e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" Nov 01 06:58:01.599975 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:01.599942 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" Nov 01 06:58:01.992404 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:01.992373 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6"] Nov 01 06:58:01.996416 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:58:01.996384 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42d21f9_d165_4278_b571_678356d3906e.slice/crio-4854f81c4b208d4bb4ff16e514875db7f4982afd77705341481231a0257fb43c WatchSource:0}: Error finding container 4854f81c4b208d4bb4ff16e514875db7f4982afd77705341481231a0257fb43c: Status 404 returned error can't find the container with id 4854f81c4b208d4bb4ff16e514875db7f4982afd77705341481231a0257fb43c Nov 01 06:58:01.998004 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:01.997983 4385 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:58:02.178674 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:02.178642 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" event={"ID":"c42d21f9-d165-4278-b571-678356d3906e","Type":"ContainerStarted","Data":"4854f81c4b208d4bb4ff16e514875db7f4982afd77705341481231a0257fb43c"} Nov 01 06:58:03.188015 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:03.187930 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" event={"ID":"c42d21f9-d165-4278-b571-678356d3906e","Type":"ContainerStarted","Data":"30165fef155e1487a939819759bef5c02bf02712da69501fc275bc34b86523a1"} Nov 01 06:58:03.188515 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:03.188064 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" Nov 01 06:58:03.302375 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:03.302320 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" podStartSLOduration=2.459299319 podStartE2EDuration="3.302307081s" totalImagesPullingTime="843.007762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:58:03.301093841 +0000 UTC m=+2893.064893696" watchObservedRunningTime="2026-11-01 06:58:03.302307081 +0000 UTC m=+2893.066106936" Nov 01 06:58:05.416177 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.416145 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9"] Nov 01 06:58:05.420984 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.420952 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.473361 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.473327 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-3db672-openshift-default-dockercfg-dng52\"" Nov 01 06:58:05.529283 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.529251 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9"] Nov 01 06:58:05.573369 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.573337 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-credential-socket\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.573369 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.573372 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5efe5a5b-a74f-4f75-a733-619770f4790c-istiod-ca-cert\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.573612 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.573396 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-workload-certs\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.573612 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.573469 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-token\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.573612 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.573511 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-podinfo\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.573612 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.573570 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-workload-socket\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.573612 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.573589 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-envoy\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.573862 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.573646 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j54tq\" (UniqueName: \"kubernetes.io/projected/5efe5a5b-a74f-4f75-a733-619770f4790c-kube-api-access-j54tq\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.573862 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.573731 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-data\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.675489 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.675406 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-credential-socket\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.675489 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.675444 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5efe5a5b-a74f-4f75-a733-619770f4790c-istiod-ca-cert\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.675723 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.675586 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-workload-certs\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.675723 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.675620 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-token\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.675723 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.675638 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-podinfo\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.675910 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.675767 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-workload-socket\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.675910 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.675802 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-envoy\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.675910 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.675882 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-credential-socket\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.676075 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.675942 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j54tq\" (UniqueName: \"kubernetes.io/projected/5efe5a5b-a74f-4f75-a733-619770f4790c-kube-api-access-j54tq\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.676075 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.675959 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-workload-certs\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.676075 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.676004 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-data\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.676241 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.676129 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-workload-socket\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.676297 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.676257 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5efe5a5b-a74f-4f75-a733-619770f4790c-istiod-ca-cert\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.676630 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.676608 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-data\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.678069 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.678034 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-envoy\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.678297 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.678282 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-podinfo\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.731987 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.731965 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-token\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.732127 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.732032 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j54tq\" (UniqueName: \"kubernetes.io/projected/5efe5a5b-a74f-4f75-a733-619770f4790c-kube-api-access-j54tq\") pod \"e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.740002 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.739983 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:05.869440 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:05.869391 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:58:06.166408 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:58:06.166380 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5efe5a5b_a74f_4f75_a733_619770f4790c.slice/crio-9dba9fcbf10c1a3c4f08e1f0c8c5acb1f1d85a50b81ef3252438bbe1de3f22a3 WatchSource:0}: Error finding container 9dba9fcbf10c1a3c4f08e1f0c8c5acb1f1d85a50b81ef3252438bbe1de3f22a3: Status 404 returned error can't find the container with id 9dba9fcbf10c1a3c4f08e1f0c8c5acb1f1d85a50b81ef3252438bbe1de3f22a3 Nov 01 06:58:06.167412 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:06.167391 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9"] Nov 01 06:58:06.169441 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:06.169411 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:58:06.169528 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:06.169475 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:58:06.169528 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:06.169503 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 06:58:06.210072 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:06.210033 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" event={"ID":"5efe5a5b-a74f-4f75-a733-619770f4790c","Type":"ContainerStarted","Data":"9dba9fcbf10c1a3c4f08e1f0c8c5acb1f1d85a50b81ef3252438bbe1de3f22a3"} Nov 01 06:58:06.784061 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:58:06.784018 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:58:07.219947 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:07.219911 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" event={"ID":"5efe5a5b-a74f-4f75-a733-619770f4790c","Type":"ContainerStarted","Data":"48860f04357f0c4a9a02c116f62113329a6e3163606529d247979b9467b61841"} Nov 01 06:58:07.405861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:07.401960 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" podStartSLOduration=2.4019376980000002 podStartE2EDuration="2.401937698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:58:07.401242851 +0000 UTC m=+2897.165042709" watchObservedRunningTime="2026-11-01 06:58:07.401937698 +0000 UTC m=+2897.165737556" Nov 01 06:58:07.741103 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:07.741069 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:07.743055 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:07.743023 4385 patch_prober.go:29] interesting pod/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 01 06:58:07.743191 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:07.743073 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" podUID="5efe5a5b-a74f-4f75-a733-619770f4790c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 01 06:58:08.741106 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:08.741070 4385 patch_prober.go:29] interesting pod/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 01 06:58:08.741585 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:08.741127 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" podUID="5efe5a5b-a74f-4f75-a733-619770f4790c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 01 06:58:09.211858 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:09.211297 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" Nov 01 06:58:09.741445 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:09.741405 4385 patch_prober.go:29] interesting pod/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 01 06:58:09.741861 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:09.741480 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" podUID="5efe5a5b-a74f-4f75-a733-619770f4790c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 01 06:58:10.667441 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:10.667086 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:58:10.740994 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:10.740963 4385 patch_prober.go:29] interesting pod/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 01 06:58:10.741198 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:10.741014 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" podUID="5efe5a5b-a74f-4f75-a733-619770f4790c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 01 06:58:11.740733 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:11.740697 4385 patch_prober.go:29] interesting pod/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 01 06:58:11.741184 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:11.740755 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" podUID="5efe5a5b-a74f-4f75-a733-619770f4790c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 01 06:58:12.746677 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:12.746646 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:12.747097 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:12.746955 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:12.747656 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:12.747638 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 06:58:15.330759 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:15.330718 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:58:15.519107 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:15.519076 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:58:15.664133 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:15.664052 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:58:17.906853 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:17.906808 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:58:18.055786 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:18.055748 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:58:20.175631 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:20.175594 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:58:24.858201 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:24.858165 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:58:29.505394 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:29.505346 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:58:34.332722 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:34.332644 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:58:36.781198 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:58:36.781157 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:58:39.144155 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:39.144117 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:58:43.994202 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:43.994154 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:58:48.778124 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:48.778088 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:58:50.442869 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:50.442801 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:58:53.419714 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:53.419683 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:58:58.109166 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:58:58.109123 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:02.834125 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:02.834088 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:06.782771 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:59:06.782731 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:59:07.636909 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:07.636874 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:12.546894 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:12.546860 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:12.687977 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:12.687947 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:13.090060 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:13.090014 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:13.756925 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:13.756865 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:14.085773 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:14.085741 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:17.388664 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:17.388631 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:17.635731 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:17.635692 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:17.925965 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:17.925922 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:18.081246 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:18.081213 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:18.631039 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:18.631011 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:20.813667 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:20.813624 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:20.990045 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:20.990002 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:21.505554 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:21.505521 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:21.539479 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:21.539443 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7"] Nov 01 06:59:21.545195 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:21.545166 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7" Nov 01 06:59:21.599782 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:21.599750 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7"] Nov 01 06:59:21.643918 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:21.643890 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c52bfa\"" Nov 01 06:59:21.644117 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:21.643955 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c52bfa-dockercfg-k58w4\"" Nov 01 06:59:21.660142 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:21.660108 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:21.680733 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:21.680704 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nrf7\" (UniqueName: \"kubernetes.io/projected/0a880bf2-33b3-4d7a-adfa-ed245baa4f9f-kube-api-access-5nrf7\") pod \"payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7\" (UID: \"0a880bf2-33b3-4d7a-adfa-ed245baa4f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7" Nov 01 06:59:21.680918 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:21.680763 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a880bf2-33b3-4d7a-adfa-ed245baa4f9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7\" (UID: \"0a880bf2-33b3-4d7a-adfa-ed245baa4f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7" Nov 01 06:59:21.782246 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:21.782153 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nrf7\" (UniqueName: \"kubernetes.io/projected/0a880bf2-33b3-4d7a-adfa-ed245baa4f9f-kube-api-access-5nrf7\") pod \"payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7\" (UID: \"0a880bf2-33b3-4d7a-adfa-ed245baa4f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7" Nov 01 06:59:21.782246 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:21.782213 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a880bf2-33b3-4d7a-adfa-ed245baa4f9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7\" (UID: \"0a880bf2-33b3-4d7a-adfa-ed245baa4f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7" Nov 01 06:59:21.783156 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:21.783133 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a880bf2-33b3-4d7a-adfa-ed245baa4f9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7\" (UID: \"0a880bf2-33b3-4d7a-adfa-ed245baa4f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7" Nov 01 06:59:21.838925 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:21.838895 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nrf7\" (UniqueName: \"kubernetes.io/projected/0a880bf2-33b3-4d7a-adfa-ed245baa4f9f-kube-api-access-5nrf7\") pod \"payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7\" (UID: \"0a880bf2-33b3-4d7a-adfa-ed245baa4f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7" Nov 01 06:59:21.863797 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:21.863767 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7" Nov 01 06:59:22.272828 ip-10-0-140-68 kubenswrapper[4385]: W1101 06:59:22.272795 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a880bf2_33b3_4d7a_adfa_ed245baa4f9f.slice/crio-ee59b17a444d8b1f334453117f1594d97fbc38dd7a63b042b1f0684dc201a625 WatchSource:0}: Error finding container ee59b17a444d8b1f334453117f1594d97fbc38dd7a63b042b1f0684dc201a625: Status 404 returned error can't find the container with id ee59b17a444d8b1f334453117f1594d97fbc38dd7a63b042b1f0684dc201a625 Nov 01 06:59:22.273584 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:22.273552 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7"] Nov 01 06:59:22.414792 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:22.414758 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:22.787911 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:22.787872 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7" event={"ID":"0a880bf2-33b3-4d7a-adfa-ed245baa4f9f","Type":"ContainerStarted","Data":"be2cca507693adba475bb0aa0dae6f322898904d5db1c256023cf0fe6b33e33e"} Nov 01 06:59:22.788115 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:22.787931 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7" event={"ID":"0a880bf2-33b3-4d7a-adfa-ed245baa4f9f","Type":"ContainerStarted","Data":"ee59b17a444d8b1f334453117f1594d97fbc38dd7a63b042b1f0684dc201a625"} Nov 01 06:59:22.788115 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:22.787960 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7" Nov 01 06:59:22.901224 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:22.900988 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7" podStartSLOduration=1.900970184 podStartE2EDuration="1.900970184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:59:22.900275783 +0000 UTC m=+2972.664075639" watchObservedRunningTime="2026-11-01 06:59:22.900970184 +0000 UTC m=+2972.664770042" Nov 01 06:59:27.136864 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:27.136811 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:32.013602 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:32.013571 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:33.790573 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:33.790534 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:33.797782 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:33.797761 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7" Nov 01 06:59:36.801855 ip-10-0-140-68 kubenswrapper[4385]: E1101 06:59:36.798981 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 06:59:36.997927 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:36.997890 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:37.814296 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:37.814242 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:41.676911 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:41.676873 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:46.424105 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:46.424068 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:49.470499 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:49.470458 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:50.779266 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:50.779234 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:51.423132 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:51.423098 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 06:59:56.121313 ip-10-0-140-68 kubenswrapper[4385]: I1101 06:59:56.121278 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:00.208238 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.208201 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk"] Nov 01 07:00:00.212486 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.212463 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv"] Nov 01 07:00:00.212616 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.212572 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" Nov 01 07:00:00.216598 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.216578 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d"] Nov 01 07:00:00.216720 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.216672 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" Nov 01 07:00:00.221049 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.220728 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d" Nov 01 07:00:00.257938 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.257911 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s"] Nov 01 07:00:00.261844 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.261820 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp"] Nov 01 07:00:00.261980 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.261945 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" Nov 01 07:00:00.266778 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.265992 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m"] Nov 01 07:00:00.266778 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.266173 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" Nov 01 07:00:00.270141 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.270119 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg"] Nov 01 07:00:00.270283 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.270264 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" Nov 01 07:00:00.274563 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.274545 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" Nov 01 07:00:00.346097 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.346046 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv"] Nov 01 07:00:00.350164 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.350134 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d"] Nov 01 07:00:00.350445 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.350423 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\"" Nov 01 07:00:00.355043 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.355020 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr2lc\" (UniqueName: \"kubernetes.io/projected/60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac-kube-api-access-qr2lc\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d\" (UID: \"60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d" Nov 01 07:00:00.355158 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.355061 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3871d20b-6d9f-4c6e-87b1-8e66cc4ef350-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv\" (UID: \"3871d20b-6d9f-4c6e-87b1-8e66cc4ef350\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" Nov 01 07:00:00.355158 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.355086 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktgl7\" (UniqueName: \"kubernetes.io/projected/d27e5671-8155-4605-9333-001d14195f62-kube-api-access-ktgl7\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk\" (UID: \"d27e5671-8155-4605-9333-001d14195f62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" Nov 01 07:00:00.355287 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.355173 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rxn6\" (UniqueName: \"kubernetes.io/projected/3871d20b-6d9f-4c6e-87b1-8e66cc4ef350-kube-api-access-6rxn6\") pod \"maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv\" (UID: \"3871d20b-6d9f-4c6e-87b1-8e66cc4ef350\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" Nov 01 07:00:00.355287 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.355206 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d27e5671-8155-4605-9333-001d14195f62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk\" (UID: \"d27e5671-8155-4605-9333-001d14195f62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" Nov 01 07:00:00.355400 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.355370 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d\" (UID: \"60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d" Nov 01 07:00:00.391504 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.391470 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m"] Nov 01 07:00:00.391679 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.391527 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg"] Nov 01 07:00:00.393274 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.393246 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp"] Nov 01 07:00:00.411346 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.411314 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s"] Nov 01 07:00:00.411503 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.411384 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk"] Nov 01 07:00:00.456431 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.456399 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rxn6\" (UniqueName: \"kubernetes.io/projected/3871d20b-6d9f-4c6e-87b1-8e66cc4ef350-kube-api-access-6rxn6\") pod \"maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv\" (UID: \"3871d20b-6d9f-4c6e-87b1-8e66cc4ef350\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" Nov 01 07:00:00.456431 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.456442 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9debfa95-4fbc-4962-8177-6a53821135ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp\" (UID: \"9debfa95-4fbc-4962-8177-6a53821135ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" Nov 01 07:00:00.456655 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.456466 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nhvm\" (UniqueName: \"kubernetes.io/projected/9debfa95-4fbc-4962-8177-6a53821135ca-kube-api-access-6nhvm\") pod \"maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp\" (UID: \"9debfa95-4fbc-4962-8177-6a53821135ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" Nov 01 07:00:00.456655 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.456537 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247e6488-ee32-4931-9670-ea759c4cc896-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s\" (UID: \"247e6488-ee32-4931-9670-ea759c4cc896\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" Nov 01 07:00:00.456655 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.456565 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/becaf42b-35ef-46db-a4f8-0cc8780722f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg\" (UID: \"becaf42b-35ef-46db-a4f8-0cc8780722f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" Nov 01 07:00:00.456655 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.456609 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d27e5671-8155-4605-9333-001d14195f62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk\" (UID: \"d27e5671-8155-4605-9333-001d14195f62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" Nov 01 07:00:00.456783 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.456674 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b1e9b1-6c86-4818-bab1-715cad35fb8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m\" (UID: \"f3b1e9b1-6c86-4818-bab1-715cad35fb8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" Nov 01 07:00:00.456828 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.456811 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwffc\" (UniqueName: \"kubernetes.io/projected/f3b1e9b1-6c86-4818-bab1-715cad35fb8c-kube-api-access-xwffc\") pod \"maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m\" (UID: \"f3b1e9b1-6c86-4818-bab1-715cad35fb8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" Nov 01 07:00:00.456931 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.456912 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d\" (UID: \"60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d" Nov 01 07:00:00.456986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.456974 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgd7r\" (UniqueName: \"kubernetes.io/projected/becaf42b-35ef-46db-a4f8-0cc8780722f0-kube-api-access-pgd7r\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg\" (UID: \"becaf42b-35ef-46db-a4f8-0cc8780722f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" Nov 01 07:00:00.457063 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.457048 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr2lc\" (UniqueName: \"kubernetes.io/projected/60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac-kube-api-access-qr2lc\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d\" (UID: \"60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d" Nov 01 07:00:00.457117 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.457093 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3871d20b-6d9f-4c6e-87b1-8e66cc4ef350-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv\" (UID: \"3871d20b-6d9f-4c6e-87b1-8e66cc4ef350\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" Nov 01 07:00:00.457195 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.457126 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktgl7\" (UniqueName: \"kubernetes.io/projected/d27e5671-8155-4605-9333-001d14195f62-kube-api-access-ktgl7\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk\" (UID: \"d27e5671-8155-4605-9333-001d14195f62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" Nov 01 07:00:00.457195 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.457163 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz9jp\" (UniqueName: \"kubernetes.io/projected/247e6488-ee32-4931-9670-ea759c4cc896-kube-api-access-zz9jp\") pod \"maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s\" (UID: \"247e6488-ee32-4931-9670-ea759c4cc896\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" Nov 01 07:00:00.457486 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.457465 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d27e5671-8155-4605-9333-001d14195f62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk\" (UID: \"d27e5671-8155-4605-9333-001d14195f62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" Nov 01 07:00:00.457643 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.457627 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d\" (UID: \"60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d" Nov 01 07:00:00.457872 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.457823 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3871d20b-6d9f-4c6e-87b1-8e66cc4ef350-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv\" (UID: \"3871d20b-6d9f-4c6e-87b1-8e66cc4ef350\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" Nov 01 07:00:00.512967 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.512899 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rxn6\" (UniqueName: \"kubernetes.io/projected/3871d20b-6d9f-4c6e-87b1-8e66cc4ef350-kube-api-access-6rxn6\") pod \"maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv\" (UID: \"3871d20b-6d9f-4c6e-87b1-8e66cc4ef350\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" Nov 01 07:00:00.513102 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.513087 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktgl7\" (UniqueName: \"kubernetes.io/projected/d27e5671-8155-4605-9333-001d14195f62-kube-api-access-ktgl7\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk\" (UID: \"d27e5671-8155-4605-9333-001d14195f62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" Nov 01 07:00:00.513263 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.513242 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr2lc\" (UniqueName: \"kubernetes.io/projected/60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac-kube-api-access-qr2lc\") pod \"maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d\" (UID: \"60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d" Nov 01 07:00:00.528728 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.528624 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" Nov 01 07:00:00.536248 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.536226 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" Nov 01 07:00:00.541554 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.541525 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d" Nov 01 07:00:00.561709 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.561674 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz9jp\" (UniqueName: \"kubernetes.io/projected/247e6488-ee32-4931-9670-ea759c4cc896-kube-api-access-zz9jp\") pod \"maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s\" (UID: \"247e6488-ee32-4931-9670-ea759c4cc896\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" Nov 01 07:00:00.561875 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.561781 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9debfa95-4fbc-4962-8177-6a53821135ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp\" (UID: \"9debfa95-4fbc-4962-8177-6a53821135ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" Nov 01 07:00:00.561875 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.561819 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nhvm\" (UniqueName: \"kubernetes.io/projected/9debfa95-4fbc-4962-8177-6a53821135ca-kube-api-access-6nhvm\") pod \"maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp\" (UID: \"9debfa95-4fbc-4962-8177-6a53821135ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" Nov 01 07:00:00.562003 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.561881 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247e6488-ee32-4931-9670-ea759c4cc896-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s\" (UID: \"247e6488-ee32-4931-9670-ea759c4cc896\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" Nov 01 07:00:00.562003 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.561919 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/becaf42b-35ef-46db-a4f8-0cc8780722f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg\" (UID: \"becaf42b-35ef-46db-a4f8-0cc8780722f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" Nov 01 07:00:00.562141 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.562043 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b1e9b1-6c86-4818-bab1-715cad35fb8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m\" (UID: \"f3b1e9b1-6c86-4818-bab1-715cad35fb8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" Nov 01 07:00:00.562141 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.562106 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwffc\" (UniqueName: \"kubernetes.io/projected/f3b1e9b1-6c86-4818-bab1-715cad35fb8c-kube-api-access-xwffc\") pod \"maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m\" (UID: \"f3b1e9b1-6c86-4818-bab1-715cad35fb8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" Nov 01 07:00:00.562251 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.562204 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgd7r\" (UniqueName: \"kubernetes.io/projected/becaf42b-35ef-46db-a4f8-0cc8780722f0-kube-api-access-pgd7r\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg\" (UID: \"becaf42b-35ef-46db-a4f8-0cc8780722f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" Nov 01 07:00:00.563170 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.562826 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b1e9b1-6c86-4818-bab1-715cad35fb8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m\" (UID: \"f3b1e9b1-6c86-4818-bab1-715cad35fb8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" Nov 01 07:00:00.563170 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.563123 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9debfa95-4fbc-4962-8177-6a53821135ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp\" (UID: \"9debfa95-4fbc-4962-8177-6a53821135ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" Nov 01 07:00:00.563481 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.563277 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/becaf42b-35ef-46db-a4f8-0cc8780722f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg\" (UID: \"becaf42b-35ef-46db-a4f8-0cc8780722f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" Nov 01 07:00:00.563481 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.563348 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247e6488-ee32-4931-9670-ea759c4cc896-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s\" (UID: \"247e6488-ee32-4931-9670-ea759c4cc896\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" Nov 01 07:00:00.622737 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.619692 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz9jp\" (UniqueName: \"kubernetes.io/projected/247e6488-ee32-4931-9670-ea759c4cc896-kube-api-access-zz9jp\") pod \"maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s\" (UID: \"247e6488-ee32-4931-9670-ea759c4cc896\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" Nov 01 07:00:00.633668 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.633636 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nhvm\" (UniqueName: \"kubernetes.io/projected/9debfa95-4fbc-4962-8177-6a53821135ca-kube-api-access-6nhvm\") pod \"maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp\" (UID: \"9debfa95-4fbc-4962-8177-6a53821135ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" Nov 01 07:00:00.634277 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.634254 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwffc\" (UniqueName: \"kubernetes.io/projected/f3b1e9b1-6c86-4818-bab1-715cad35fb8c-kube-api-access-xwffc\") pod \"maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m\" (UID: \"f3b1e9b1-6c86-4818-bab1-715cad35fb8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" Nov 01 07:00:00.637279 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.637225 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgd7r\" (UniqueName: \"kubernetes.io/projected/becaf42b-35ef-46db-a4f8-0cc8780722f0-kube-api-access-pgd7r\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg\" (UID: \"becaf42b-35ef-46db-a4f8-0cc8780722f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" Nov 01 07:00:00.878309 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.878274 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" Nov 01 07:00:00.885750 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.885721 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" Nov 01 07:00:00.894285 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.894258 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" Nov 01 07:00:00.900354 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.900326 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" Nov 01 07:00:00.938564 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.920936 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk"] Nov 01 07:00:00.984476 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:00.984316 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d"] Nov 01 07:00:01.029854 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:01.028225 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:01.121332 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:01.121304 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv"] Nov 01 07:00:01.127806 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:01.127774 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d" event={"ID":"60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac","Type":"ContainerStarted","Data":"d1e878c11e3ab175e6a080270fd2a0eac69d6a86629bb90a0d12366830892877"} Nov 01 07:00:01.132433 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:01.132405 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" event={"ID":"d27e5671-8155-4605-9333-001d14195f62","Type":"ContainerStarted","Data":"f06b1dd07b16f8570433fe42510df2cbc650b9402b00dfba8e3a21d4fdf52edf"} Nov 01 07:00:01.304682 ip-10-0-140-68 kubenswrapper[4385]: W1101 07:00:01.304643 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247e6488_ee32_4931_9670_ea759c4cc896.slice/crio-89561fcef3b5ef13797fb437f5b5090870c235061149d8e3fed722a429732883 WatchSource:0}: Error finding container 89561fcef3b5ef13797fb437f5b5090870c235061149d8e3fed722a429732883: Status 404 returned error can't find the container with id 89561fcef3b5ef13797fb437f5b5090870c235061149d8e3fed722a429732883 Nov 01 07:00:01.306108 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:01.305198 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s"] Nov 01 07:00:01.323354 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:01.323331 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp"] Nov 01 07:00:01.324273 ip-10-0-140-68 kubenswrapper[4385]: W1101 07:00:01.324249 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9debfa95_4fbc_4962_8177_6a53821135ca.slice/crio-0de042826c21597ea82aa1f67837c3d6821783249a25547abbd5e44d87f47d20 WatchSource:0}: Error finding container 0de042826c21597ea82aa1f67837c3d6821783249a25547abbd5e44d87f47d20: Status 404 returned error can't find the container with id 0de042826c21597ea82aa1f67837c3d6821783249a25547abbd5e44d87f47d20 Nov 01 07:00:01.364039 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:01.364015 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg"] Nov 01 07:00:01.364151 ip-10-0-140-68 kubenswrapper[4385]: W1101 07:00:01.364094 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbecaf42b_35ef_46db_a4f8_0cc8780722f0.slice/crio-b1b496fb0860dbe95c09b0f6decb6f86905fd67d5d1acb6a68812b0cfb145993 WatchSource:0}: Error finding container b1b496fb0860dbe95c09b0f6decb6f86905fd67d5d1acb6a68812b0cfb145993: Status 404 returned error can't find the container with id b1b496fb0860dbe95c09b0f6decb6f86905fd67d5d1acb6a68812b0cfb145993 Nov 01 07:00:01.383028 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:01.383000 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m"] Nov 01 07:00:01.472753 ip-10-0-140-68 kubenswrapper[4385]: W1101 07:00:01.472721 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b1e9b1_6c86_4818_bab1_715cad35fb8c.slice/crio-ed0ef98723ab753c42a8455872c9ee57c4f7e62134a5ae7c20a3105c81efdc8e WatchSource:0}: Error finding container ed0ef98723ab753c42a8455872c9ee57c4f7e62134a5ae7c20a3105c81efdc8e: Status 404 returned error can't find the container with id ed0ef98723ab753c42a8455872c9ee57c4f7e62134a5ae7c20a3105c81efdc8e Nov 01 07:00:02.141794 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:02.141759 4385 generic.go:356] "Generic (PLEG): container finished" podID="60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac" containerID="22b4eee1a58eb38eb5261dfdac5b86c0e2e7fa9b7e736b896d1fda08433b1e1c" exitCode=60 Nov 01 07:00:02.141996 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:02.141871 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d" event={"ID":"60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac","Type":"ContainerDied","Data":"22b4eee1a58eb38eb5261dfdac5b86c0e2e7fa9b7e736b896d1fda08433b1e1c"} Nov 01 07:00:02.142172 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:02.142158 4385 scope.go:122] "RemoveContainer" containerID="22b4eee1a58eb38eb5261dfdac5b86c0e2e7fa9b7e736b896d1fda08433b1e1c" Nov 01 07:00:02.143547 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:02.143523 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" event={"ID":"9debfa95-4fbc-4962-8177-6a53821135ca","Type":"ContainerStarted","Data":"2f2759a43c5fc85b7909d5c7f387b50247709c44477e260cf99e2b7b05bc1756"} Nov 01 07:00:02.143661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:02.143568 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" event={"ID":"9debfa95-4fbc-4962-8177-6a53821135ca","Type":"ContainerStarted","Data":"0de042826c21597ea82aa1f67837c3d6821783249a25547abbd5e44d87f47d20"} Nov 01 07:00:02.145370 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:02.145350 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" event={"ID":"247e6488-ee32-4931-9670-ea759c4cc896","Type":"ContainerStarted","Data":"e23e9d460149dd864cfec40b6b2f19349119b9cc4570055dbe1e1d26e044a87b"} Nov 01 07:00:02.145469 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:02.145391 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" event={"ID":"247e6488-ee32-4931-9670-ea759c4cc896","Type":"ContainerStarted","Data":"89561fcef3b5ef13797fb437f5b5090870c235061149d8e3fed722a429732883"} Nov 01 07:00:02.147000 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:02.146977 4385 generic.go:356] "Generic (PLEG): container finished" podID="d27e5671-8155-4605-9333-001d14195f62" containerID="fc12d39afdcd222cb80e39eda4cc7ba520f368e6448a0ab205ea90b37a4ca399" exitCode=60 Nov 01 07:00:02.147090 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:02.147075 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" event={"ID":"d27e5671-8155-4605-9333-001d14195f62","Type":"ContainerDied","Data":"fc12d39afdcd222cb80e39eda4cc7ba520f368e6448a0ab205ea90b37a4ca399"} Nov 01 07:00:02.147327 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:02.147309 4385 scope.go:122] "RemoveContainer" containerID="fc12d39afdcd222cb80e39eda4cc7ba520f368e6448a0ab205ea90b37a4ca399" Nov 01 07:00:02.148603 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:02.148581 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" event={"ID":"f3b1e9b1-6c86-4818-bab1-715cad35fb8c","Type":"ContainerStarted","Data":"f35e6a5eac8c032608a7f9b0817a11fe79ca43c3a81ef0bf7905f709572c8ca3"} Nov 01 07:00:02.148879 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:02.148619 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" event={"ID":"f3b1e9b1-6c86-4818-bab1-715cad35fb8c","Type":"ContainerStarted","Data":"ed0ef98723ab753c42a8455872c9ee57c4f7e62134a5ae7c20a3105c81efdc8e"} Nov 01 07:00:02.150195 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:02.150171 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" event={"ID":"3871d20b-6d9f-4c6e-87b1-8e66cc4ef350","Type":"ContainerStarted","Data":"5abd60312bfba7159706ea866417fe9e8137f025319ea6f9d4dba2a564171841"} Nov 01 07:00:02.150322 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:02.150216 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" event={"ID":"3871d20b-6d9f-4c6e-87b1-8e66cc4ef350","Type":"ContainerStarted","Data":"852aed23260330c6e4deafc6cee554f0ba0e6df83eeb2c2bc75415a72960f3f4"} Nov 01 07:00:02.151569 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:02.151550 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" event={"ID":"becaf42b-35ef-46db-a4f8-0cc8780722f0","Type":"ContainerStarted","Data":"35817580d38cd080e76746ef2392258515a8f0444771868e6aa7f6113795df82"} Nov 01 07:00:02.151659 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:02.151583 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" event={"ID":"becaf42b-35ef-46db-a4f8-0cc8780722f0","Type":"ContainerStarted","Data":"b1b496fb0860dbe95c09b0f6decb6f86905fd67d5d1acb6a68812b0cfb145993"} Nov 01 07:00:02.374045 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:02.373483 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" podStartSLOduration=2.373465071 podStartE2EDuration="2.373465071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:02.368930361 +0000 UTC m=+3012.132730228" watchObservedRunningTime="2026-11-01 07:00:02.373465071 +0000 UTC m=+3012.137264927" Nov 01 07:00:02.484765 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:02.484715 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" podStartSLOduration=2.4847008329999998 podStartE2EDuration="2.484700833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:02.479782833 +0000 UTC m=+3012.243582692" watchObservedRunningTime="2026-11-01 07:00:02.484700833 +0000 UTC m=+3012.248500689" Nov 01 07:00:02.705422 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:02.705374 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" podStartSLOduration=2.705360772 podStartE2EDuration="2.705360772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:02.701284238 +0000 UTC m=+3012.465084101" watchObservedRunningTime="2026-11-01 07:00:02.705360772 +0000 UTC m=+3012.469160619" Nov 01 07:00:02.814522 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:02.814470 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" podStartSLOduration=2.8144547170000003 podStartE2EDuration="2.814454717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:02.813022721 +0000 UTC m=+3012.576822576" watchObservedRunningTime="2026-11-01 07:00:02.814454717 +0000 UTC m=+3012.578254616" Nov 01 07:00:02.926390 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:02.926338 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" podStartSLOduration=2.926322322 podStartE2EDuration="2.926322322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:02.923331815 +0000 UTC m=+3012.687131673" watchObservedRunningTime="2026-11-01 07:00:02.926322322 +0000 UTC m=+3012.690122220" Nov 01 07:00:03.166770 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:03.166727 4385 generic.go:356] "Generic (PLEG): container finished" podID="247e6488-ee32-4931-9670-ea759c4cc896" containerID="e23e9d460149dd864cfec40b6b2f19349119b9cc4570055dbe1e1d26e044a87b" exitCode=60 Nov 01 07:00:03.167000 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:03.166814 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" event={"ID":"247e6488-ee32-4931-9670-ea759c4cc896","Type":"ContainerDied","Data":"e23e9d460149dd864cfec40b6b2f19349119b9cc4570055dbe1e1d26e044a87b"} Nov 01 07:00:03.167204 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:03.167170 4385 scope.go:122] "RemoveContainer" containerID="e23e9d460149dd864cfec40b6b2f19349119b9cc4570055dbe1e1d26e044a87b" Nov 01 07:00:03.169307 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:03.169280 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" event={"ID":"d27e5671-8155-4605-9333-001d14195f62","Type":"ContainerStarted","Data":"a777718b579604ddc4995191aada303f492a6093ecf8fe75474528249f0e4d38"} Nov 01 07:00:03.171847 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:03.171806 4385 generic.go:356] "Generic (PLEG): container finished" podID="f3b1e9b1-6c86-4818-bab1-715cad35fb8c" containerID="f35e6a5eac8c032608a7f9b0817a11fe79ca43c3a81ef0bf7905f709572c8ca3" exitCode=60 Nov 01 07:00:03.171943 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:03.171881 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" event={"ID":"f3b1e9b1-6c86-4818-bab1-715cad35fb8c","Type":"ContainerDied","Data":"f35e6a5eac8c032608a7f9b0817a11fe79ca43c3a81ef0bf7905f709572c8ca3"} Nov 01 07:00:03.172575 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:03.172214 4385 scope.go:122] "RemoveContainer" containerID="f35e6a5eac8c032608a7f9b0817a11fe79ca43c3a81ef0bf7905f709572c8ca3" Nov 01 07:00:03.174744 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:03.174686 4385 generic.go:356] "Generic (PLEG): container finished" podID="3871d20b-6d9f-4c6e-87b1-8e66cc4ef350" containerID="5abd60312bfba7159706ea866417fe9e8137f025319ea6f9d4dba2a564171841" exitCode=60 Nov 01 07:00:03.174970 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:03.174787 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" event={"ID":"3871d20b-6d9f-4c6e-87b1-8e66cc4ef350","Type":"ContainerDied","Data":"5abd60312bfba7159706ea866417fe9e8137f025319ea6f9d4dba2a564171841"} Nov 01 07:00:03.175135 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:03.175118 4385 scope.go:122] "RemoveContainer" containerID="5abd60312bfba7159706ea866417fe9e8137f025319ea6f9d4dba2a564171841" Nov 01 07:00:03.177536 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:03.177513 4385 generic.go:356] "Generic (PLEG): container finished" podID="becaf42b-35ef-46db-a4f8-0cc8780722f0" containerID="35817580d38cd080e76746ef2392258515a8f0444771868e6aa7f6113795df82" exitCode=60 Nov 01 07:00:03.177666 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:03.177628 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" event={"ID":"becaf42b-35ef-46db-a4f8-0cc8780722f0","Type":"ContainerDied","Data":"35817580d38cd080e76746ef2392258515a8f0444771868e6aa7f6113795df82"} Nov 01 07:00:03.178042 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:03.178017 4385 scope.go:122] "RemoveContainer" containerID="35817580d38cd080e76746ef2392258515a8f0444771868e6aa7f6113795df82" Nov 01 07:00:03.180713 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:03.180688 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d" event={"ID":"60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac","Type":"ContainerStarted","Data":"097b5c4396fcf8dadfa0b16eac0d8369d067ab6a55ba837e7619664eb3091f4f"} Nov 01 07:00:03.189374 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:03.189341 4385 generic.go:356] "Generic (PLEG): container finished" podID="9debfa95-4fbc-4962-8177-6a53821135ca" containerID="2f2759a43c5fc85b7909d5c7f387b50247709c44477e260cf99e2b7b05bc1756" exitCode=60 Nov 01 07:00:03.189469 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:03.189408 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" event={"ID":"9debfa95-4fbc-4962-8177-6a53821135ca","Type":"ContainerDied","Data":"2f2759a43c5fc85b7909d5c7f387b50247709c44477e260cf99e2b7b05bc1756"} Nov 01 07:00:03.190141 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:03.189980 4385 scope.go:122] "RemoveContainer" containerID="2f2759a43c5fc85b7909d5c7f387b50247709c44477e260cf99e2b7b05bc1756" Nov 01 07:00:03.617387 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:03.617334 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" podStartSLOduration=3.617317517 podStartE2EDuration="3.617317517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:03.613133202 +0000 UTC m=+3013.376933058" watchObservedRunningTime="2026-11-01 07:00:03.617317517 +0000 UTC m=+3013.381117378" Nov 01 07:00:03.726876 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:03.726796 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d" podStartSLOduration=3.726782919 podStartE2EDuration="3.726782919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:03.724598489 +0000 UTC m=+3013.488398344" watchObservedRunningTime="2026-11-01 07:00:03.726782919 +0000 UTC m=+3013.490582774" Nov 01 07:00:04.199159 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:04.199121 4385 generic.go:356] "Generic (PLEG): container finished" podID="60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac" containerID="097b5c4396fcf8dadfa0b16eac0d8369d067ab6a55ba837e7619664eb3091f4f" exitCode=60 Nov 01 07:00:04.199335 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:04.199169 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d" event={"ID":"60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac","Type":"ContainerDied","Data":"097b5c4396fcf8dadfa0b16eac0d8369d067ab6a55ba837e7619664eb3091f4f"} Nov 01 07:00:04.199335 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:04.199231 4385 scope.go:122] "RemoveContainer" containerID="22b4eee1a58eb38eb5261dfdac5b86c0e2e7fa9b7e736b896d1fda08433b1e1c" Nov 01 07:00:04.199514 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:04.199500 4385 scope.go:122] "RemoveContainer" containerID="097b5c4396fcf8dadfa0b16eac0d8369d067ab6a55ba837e7619664eb3091f4f" Nov 01 07:00:04.199852 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:04.199807 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d_odh-ai-gateway-infra(60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d" podUID="60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac" Nov 01 07:00:04.201705 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:04.201682 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" event={"ID":"9debfa95-4fbc-4962-8177-6a53821135ca","Type":"ContainerStarted","Data":"634f0f1ffcf8b9d50513576cda329fcdfe1a1d54081be5c05a7f758b4c5b7bd0"} Nov 01 07:00:04.203674 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:04.203646 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" event={"ID":"247e6488-ee32-4931-9670-ea759c4cc896","Type":"ContainerStarted","Data":"506f848e4427f485d62b4b44932dd6397dc53df1e0d121df3c371ca7e489f137"} Nov 01 07:00:04.205994 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:04.205967 4385 generic.go:356] "Generic (PLEG): container finished" podID="d27e5671-8155-4605-9333-001d14195f62" containerID="a777718b579604ddc4995191aada303f492a6093ecf8fe75474528249f0e4d38" exitCode=60 Nov 01 07:00:04.206127 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:04.206051 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" event={"ID":"d27e5671-8155-4605-9333-001d14195f62","Type":"ContainerDied","Data":"a777718b579604ddc4995191aada303f492a6093ecf8fe75474528249f0e4d38"} Nov 01 07:00:04.206331 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:04.206313 4385 scope.go:122] "RemoveContainer" containerID="a777718b579604ddc4995191aada303f492a6093ecf8fe75474528249f0e4d38" Nov 01 07:00:04.206560 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:04.206543 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk_odh-ai-gateway-infra(d27e5671-8155-4605-9333-001d14195f62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" podUID="d27e5671-8155-4605-9333-001d14195f62" Nov 01 07:00:04.208953 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:04.208931 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" event={"ID":"f3b1e9b1-6c86-4818-bab1-715cad35fb8c","Type":"ContainerStarted","Data":"8f40b074767e49d29ecd0368fb10bebb8f3ac7844dacc78931c66e2e6aec11c7"} Nov 01 07:00:04.210769 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:04.210746 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" event={"ID":"3871d20b-6d9f-4c6e-87b1-8e66cc4ef350","Type":"ContainerStarted","Data":"0f62ca9390b5e40d6886d2e2093f7817c8388ef2d41720c46c708da1d3461de5"} Nov 01 07:00:04.212710 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:04.212686 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" event={"ID":"becaf42b-35ef-46db-a4f8-0cc8780722f0","Type":"ContainerStarted","Data":"b0aaf99d71641997eaea03992a6982cec1c5737224fa17a47225da4206912fa4"} Nov 01 07:00:04.214897 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:04.214875 4385 scope.go:122] "RemoveContainer" containerID="fc12d39afdcd222cb80e39eda4cc7ba520f368e6448a0ab205ea90b37a4ca399" Nov 01 07:00:05.221746 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.221715 4385 scope.go:122] "RemoveContainer" containerID="097b5c4396fcf8dadfa0b16eac0d8369d067ab6a55ba837e7619664eb3091f4f" Nov 01 07:00:05.222330 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:05.222043 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d_odh-ai-gateway-infra(60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d" podUID="60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac" Nov 01 07:00:05.223296 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.223273 4385 generic.go:356] "Generic (PLEG): container finished" podID="9debfa95-4fbc-4962-8177-6a53821135ca" containerID="634f0f1ffcf8b9d50513576cda329fcdfe1a1d54081be5c05a7f758b4c5b7bd0" exitCode=60 Nov 01 07:00:05.223416 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.223344 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" event={"ID":"9debfa95-4fbc-4962-8177-6a53821135ca","Type":"ContainerDied","Data":"634f0f1ffcf8b9d50513576cda329fcdfe1a1d54081be5c05a7f758b4c5b7bd0"} Nov 01 07:00:05.223483 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.223425 4385 scope.go:122] "RemoveContainer" containerID="2f2759a43c5fc85b7909d5c7f387b50247709c44477e260cf99e2b7b05bc1756" Nov 01 07:00:05.223540 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.223525 4385 scope.go:122] "RemoveContainer" containerID="634f0f1ffcf8b9d50513576cda329fcdfe1a1d54081be5c05a7f758b4c5b7bd0" Nov 01 07:00:05.223780 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:05.223762 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp_odh-ai-gateway-infra(9debfa95-4fbc-4962-8177-6a53821135ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" podUID="9debfa95-4fbc-4962-8177-6a53821135ca" Nov 01 07:00:05.225409 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.225390 4385 generic.go:356] "Generic (PLEG): container finished" podID="247e6488-ee32-4931-9670-ea759c4cc896" containerID="506f848e4427f485d62b4b44932dd6397dc53df1e0d121df3c371ca7e489f137" exitCode=60 Nov 01 07:00:05.225516 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.225481 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" event={"ID":"247e6488-ee32-4931-9670-ea759c4cc896","Type":"ContainerDied","Data":"506f848e4427f485d62b4b44932dd6397dc53df1e0d121df3c371ca7e489f137"} Nov 01 07:00:05.225650 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.225634 4385 scope.go:122] "RemoveContainer" containerID="506f848e4427f485d62b4b44932dd6397dc53df1e0d121df3c371ca7e489f137" Nov 01 07:00:05.225954 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:05.225925 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s_odh-ai-gateway-infra(247e6488-ee32-4931-9670-ea759c4cc896)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" podUID="247e6488-ee32-4931-9670-ea759c4cc896" Nov 01 07:00:05.227466 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.227447 4385 scope.go:122] "RemoveContainer" containerID="a777718b579604ddc4995191aada303f492a6093ecf8fe75474528249f0e4d38" Nov 01 07:00:05.227674 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:05.227653 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk_odh-ai-gateway-infra(d27e5671-8155-4605-9333-001d14195f62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" podUID="d27e5671-8155-4605-9333-001d14195f62" Nov 01 07:00:05.229114 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.229097 4385 generic.go:356] "Generic (PLEG): container finished" podID="f3b1e9b1-6c86-4818-bab1-715cad35fb8c" containerID="8f40b074767e49d29ecd0368fb10bebb8f3ac7844dacc78931c66e2e6aec11c7" exitCode=60 Nov 01 07:00:05.229204 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.229154 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" event={"ID":"f3b1e9b1-6c86-4818-bab1-715cad35fb8c","Type":"ContainerDied","Data":"8f40b074767e49d29ecd0368fb10bebb8f3ac7844dacc78931c66e2e6aec11c7"} Nov 01 07:00:05.229384 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.229364 4385 scope.go:122] "RemoveContainer" containerID="8f40b074767e49d29ecd0368fb10bebb8f3ac7844dacc78931c66e2e6aec11c7" Nov 01 07:00:05.229632 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:05.229611 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m_odh-ai-gateway-infra(f3b1e9b1-6c86-4818-bab1-715cad35fb8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" podUID="f3b1e9b1-6c86-4818-bab1-715cad35fb8c" Nov 01 07:00:05.230992 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.230971 4385 generic.go:356] "Generic (PLEG): container finished" podID="3871d20b-6d9f-4c6e-87b1-8e66cc4ef350" containerID="0f62ca9390b5e40d6886d2e2093f7817c8388ef2d41720c46c708da1d3461de5" exitCode=60 Nov 01 07:00:05.231097 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.231061 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" event={"ID":"3871d20b-6d9f-4c6e-87b1-8e66cc4ef350","Type":"ContainerDied","Data":"0f62ca9390b5e40d6886d2e2093f7817c8388ef2d41720c46c708da1d3461de5"} Nov 01 07:00:05.231234 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.231214 4385 scope.go:122] "RemoveContainer" containerID="0f62ca9390b5e40d6886d2e2093f7817c8388ef2d41720c46c708da1d3461de5" Nov 01 07:00:05.231456 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:05.231422 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv_odh-ai-gateway-infra(3871d20b-6d9f-4c6e-87b1-8e66cc4ef350)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" podUID="3871d20b-6d9f-4c6e-87b1-8e66cc4ef350" Nov 01 07:00:05.233333 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.233311 4385 generic.go:356] "Generic (PLEG): container finished" podID="becaf42b-35ef-46db-a4f8-0cc8780722f0" containerID="b0aaf99d71641997eaea03992a6982cec1c5737224fa17a47225da4206912fa4" exitCode=60 Nov 01 07:00:05.233433 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.233342 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" event={"ID":"becaf42b-35ef-46db-a4f8-0cc8780722f0","Type":"ContainerDied","Data":"b0aaf99d71641997eaea03992a6982cec1c5737224fa17a47225da4206912fa4"} Nov 01 07:00:05.233537 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.233524 4385 scope.go:122] "RemoveContainer" containerID="b0aaf99d71641997eaea03992a6982cec1c5737224fa17a47225da4206912fa4" Nov 01 07:00:05.233705 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:05.233689 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg_odh-ai-gateway-infra(becaf42b-35ef-46db-a4f8-0cc8780722f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" podUID="becaf42b-35ef-46db-a4f8-0cc8780722f0" Nov 01 07:00:05.238301 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.238283 4385 scope.go:122] "RemoveContainer" containerID="e23e9d460149dd864cfec40b6b2f19349119b9cc4570055dbe1e1d26e044a87b" Nov 01 07:00:05.251946 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.251929 4385 scope.go:122] "RemoveContainer" containerID="f35e6a5eac8c032608a7f9b0817a11fe79ca43c3a81ef0bf7905f709572c8ca3" Nov 01 07:00:05.264560 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.264542 4385 scope.go:122] "RemoveContainer" containerID="5abd60312bfba7159706ea866417fe9e8137f025319ea6f9d4dba2a564171841" Nov 01 07:00:05.280660 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.280643 4385 scope.go:122] "RemoveContainer" containerID="35817580d38cd080e76746ef2392258515a8f0444771868e6aa7f6113795df82" Nov 01 07:00:05.769884 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:05.769844 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:06.247066 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:06.247037 4385 scope.go:122] "RemoveContainer" containerID="634f0f1ffcf8b9d50513576cda329fcdfe1a1d54081be5c05a7f758b4c5b7bd0" Nov 01 07:00:06.247529 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:06.247329 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp_odh-ai-gateway-infra(9debfa95-4fbc-4962-8177-6a53821135ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" podUID="9debfa95-4fbc-4962-8177-6a53821135ca" Nov 01 07:00:06.249003 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:06.248984 4385 scope.go:122] "RemoveContainer" containerID="506f848e4427f485d62b4b44932dd6397dc53df1e0d121df3c371ca7e489f137" Nov 01 07:00:06.249216 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:06.249198 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s_odh-ai-gateway-infra(247e6488-ee32-4931-9670-ea759c4cc896)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" podUID="247e6488-ee32-4931-9670-ea759c4cc896" Nov 01 07:00:06.250982 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:06.250959 4385 scope.go:122] "RemoveContainer" containerID="8f40b074767e49d29ecd0368fb10bebb8f3ac7844dacc78931c66e2e6aec11c7" Nov 01 07:00:06.251221 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:06.251198 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m_odh-ai-gateway-infra(f3b1e9b1-6c86-4818-bab1-715cad35fb8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" podUID="f3b1e9b1-6c86-4818-bab1-715cad35fb8c" Nov 01 07:00:06.252484 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:06.252469 4385 scope.go:122] "RemoveContainer" containerID="0f62ca9390b5e40d6886d2e2093f7817c8388ef2d41720c46c708da1d3461de5" Nov 01 07:00:06.252674 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:06.252656 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv_odh-ai-gateway-infra(3871d20b-6d9f-4c6e-87b1-8e66cc4ef350)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" podUID="3871d20b-6d9f-4c6e-87b1-8e66cc4ef350" Nov 01 07:00:06.254172 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:06.254157 4385 scope.go:122] "RemoveContainer" containerID="b0aaf99d71641997eaea03992a6982cec1c5737224fa17a47225da4206912fa4" Nov 01 07:00:06.254354 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:06.254334 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg_odh-ai-gateway-infra(becaf42b-35ef-46db-a4f8-0cc8780722f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" podUID="becaf42b-35ef-46db-a4f8-0cc8780722f0" Nov 01 07:00:06.783867 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:06.780653 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:00:09.654807 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:09.654776 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:10.503431 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:10.503393 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:15.216481 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:15.216427 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:15.377537 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:15.377503 4385 scope.go:122] "RemoveContainer" containerID="097b5c4396fcf8dadfa0b16eac0d8369d067ab6a55ba837e7619664eb3091f4f" Nov 01 07:00:15.377754 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:15.377616 4385 scope.go:122] "RemoveContainer" containerID="8f40b074767e49d29ecd0368fb10bebb8f3ac7844dacc78931c66e2e6aec11c7" Nov 01 07:00:16.347937 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:16.347898 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" event={"ID":"f3b1e9b1-6c86-4818-bab1-715cad35fb8c","Type":"ContainerStarted","Data":"f6647f88146d302121072708417e79086dd6d3d5ee677dd2207422c6b245b153"} Nov 01 07:00:16.350360 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:16.350330 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d" event={"ID":"60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac","Type":"ContainerStarted","Data":"83a54e45197f160ca7011bb9391b3652bfe614435bf0eba9b7a8f8dd3726442a"} Nov 01 07:00:16.377793 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:16.377768 4385 scope.go:122] "RemoveContainer" containerID="0f62ca9390b5e40d6886d2e2093f7817c8388ef2d41720c46c708da1d3461de5" Nov 01 07:00:17.359637 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:17.359608 4385 generic.go:356] "Generic (PLEG): container finished" podID="60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac" containerID="83a54e45197f160ca7011bb9391b3652bfe614435bf0eba9b7a8f8dd3726442a" exitCode=60 Nov 01 07:00:17.360177 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:17.359651 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d" event={"ID":"60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac","Type":"ContainerDied","Data":"83a54e45197f160ca7011bb9391b3652bfe614435bf0eba9b7a8f8dd3726442a"} Nov 01 07:00:17.360177 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:17.359729 4385 scope.go:122] "RemoveContainer" containerID="097b5c4396fcf8dadfa0b16eac0d8369d067ab6a55ba837e7619664eb3091f4f" Nov 01 07:00:17.360409 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:17.360225 4385 scope.go:122] "RemoveContainer" containerID="83a54e45197f160ca7011bb9391b3652bfe614435bf0eba9b7a8f8dd3726442a" Nov 01 07:00:17.360544 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:17.360522 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d_odh-ai-gateway-infra(60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d" podUID="60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac" Nov 01 07:00:17.365785 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:17.365745 4385 generic.go:356] "Generic (PLEG): container finished" podID="f3b1e9b1-6c86-4818-bab1-715cad35fb8c" containerID="f6647f88146d302121072708417e79086dd6d3d5ee677dd2207422c6b245b153" exitCode=60 Nov 01 07:00:17.365931 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:17.365809 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" event={"ID":"f3b1e9b1-6c86-4818-bab1-715cad35fb8c","Type":"ContainerDied","Data":"f6647f88146d302121072708417e79086dd6d3d5ee677dd2207422c6b245b153"} Nov 01 07:00:17.366227 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:17.366213 4385 scope.go:122] "RemoveContainer" containerID="f6647f88146d302121072708417e79086dd6d3d5ee677dd2207422c6b245b153" Nov 01 07:00:17.366581 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:17.366545 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m_odh-ai-gateway-infra(f3b1e9b1-6c86-4818-bab1-715cad35fb8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" podUID="f3b1e9b1-6c86-4818-bab1-715cad35fb8c" Nov 01 07:00:17.368285 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:17.368251 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" event={"ID":"3871d20b-6d9f-4c6e-87b1-8e66cc4ef350","Type":"ContainerStarted","Data":"9f4ff76eb3e87a917f2d24b37a6db7c9030407b6988f1f6e7bf4100d537ece5b"} Nov 01 07:00:17.375011 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:17.374979 4385 scope.go:122] "RemoveContainer" containerID="8f40b074767e49d29ecd0368fb10bebb8f3ac7844dacc78931c66e2e6aec11c7" Nov 01 07:00:17.377433 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:17.377412 4385 scope.go:122] "RemoveContainer" containerID="634f0f1ffcf8b9d50513576cda329fcdfe1a1d54081be5c05a7f758b4c5b7bd0" Nov 01 07:00:17.378264 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:17.378245 4385 scope.go:122] "RemoveContainer" containerID="a777718b579604ddc4995191aada303f492a6093ecf8fe75474528249f0e4d38" Nov 01 07:00:17.378460 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:17.378444 4385 scope.go:122] "RemoveContainer" containerID="b0aaf99d71641997eaea03992a6982cec1c5737224fa17a47225da4206912fa4" Nov 01 07:00:17.477074 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:17.477041 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m"] Nov 01 07:00:17.584308 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:17.584285 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d"] Nov 01 07:00:18.377444 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.377420 4385 scope.go:122] "RemoveContainer" containerID="506f848e4427f485d62b4b44932dd6397dc53df1e0d121df3c371ca7e489f137" Nov 01 07:00:18.389983 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.389952 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" event={"ID":"9debfa95-4fbc-4962-8177-6a53821135ca","Type":"ContainerStarted","Data":"4e02550c512b8435f174f7a6836638914be82c742d278f96b08f52f47d12654b"} Nov 01 07:00:18.390110 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.390024 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" event={"ID":"d27e5671-8155-4605-9333-001d14195f62","Type":"ContainerStarted","Data":"f7782e866517460d9715e58ed19fa4e863a5b73dd0532f0f2a772a7e0e62a847"} Nov 01 07:00:18.390445 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.390427 4385 scope.go:122] "RemoveContainer" containerID="f7782e866517460d9715e58ed19fa4e863a5b73dd0532f0f2a772a7e0e62a847" Nov 01 07:00:18.390902 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:18.390728 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk_odh-ai-gateway-infra(d27e5671-8155-4605-9333-001d14195f62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" podUID="d27e5671-8155-4605-9333-001d14195f62" Nov 01 07:00:18.393847 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.393811 4385 generic.go:356] "Generic (PLEG): container finished" podID="3871d20b-6d9f-4c6e-87b1-8e66cc4ef350" containerID="9f4ff76eb3e87a917f2d24b37a6db7c9030407b6988f1f6e7bf4100d537ece5b" exitCode=60 Nov 01 07:00:18.393942 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.393885 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" event={"ID":"3871d20b-6d9f-4c6e-87b1-8e66cc4ef350","Type":"ContainerDied","Data":"9f4ff76eb3e87a917f2d24b37a6db7c9030407b6988f1f6e7bf4100d537ece5b"} Nov 01 07:00:18.394020 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.393942 4385 scope.go:122] "RemoveContainer" containerID="0f62ca9390b5e40d6886d2e2093f7817c8388ef2d41720c46c708da1d3461de5" Nov 01 07:00:18.394473 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.394329 4385 scope.go:122] "RemoveContainer" containerID="9f4ff76eb3e87a917f2d24b37a6db7c9030407b6988f1f6e7bf4100d537ece5b" Nov 01 07:00:18.394609 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:18.394586 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv_odh-ai-gateway-infra(3871d20b-6d9f-4c6e-87b1-8e66cc4ef350)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" podUID="3871d20b-6d9f-4c6e-87b1-8e66cc4ef350" Nov 01 07:00:18.405194 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.405154 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" event={"ID":"becaf42b-35ef-46db-a4f8-0cc8780722f0","Type":"ContainerStarted","Data":"654128d2b576ef11d923adc82c940ec8f8ed63ee51e04c6c0efc510114ce4bb8"} Nov 01 07:00:18.405491 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.405476 4385 scope.go:122] "RemoveContainer" containerID="654128d2b576ef11d923adc82c940ec8f8ed63ee51e04c6c0efc510114ce4bb8" Nov 01 07:00:18.405757 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:18.405738 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg_odh-ai-gateway-infra(becaf42b-35ef-46db-a4f8-0cc8780722f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" podUID="becaf42b-35ef-46db-a4f8-0cc8780722f0" Nov 01 07:00:18.629235 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.629178 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d" Nov 01 07:00:18.631209 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.631191 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" Nov 01 07:00:18.706770 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.706735 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b1e9b1-6c86-4818-bab1-715cad35fb8c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b1e9b1-6c86-4818-bab1-715cad35fb8c-maas-api-service-ca\") pod \"f3b1e9b1-6c86-4818-bab1-715cad35fb8c\" (UID: \"f3b1e9b1-6c86-4818-bab1-715cad35fb8c\") " Nov 01 07:00:18.706992 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.706931 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac-maas-api-service-ca\") pod \"60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac\" (UID: \"60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac\") " Nov 01 07:00:18.706992 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.706978 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b1e9b1-6c86-4818-bab1-715cad35fb8c-kube-api-access-xwffc\" (UniqueName: \"kubernetes.io/projected/f3b1e9b1-6c86-4818-bab1-715cad35fb8c-kube-api-access-xwffc\") pod \"f3b1e9b1-6c86-4818-bab1-715cad35fb8c\" (UID: \"f3b1e9b1-6c86-4818-bab1-715cad35fb8c\") " Nov 01 07:00:18.707124 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.707073 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac-kube-api-access-qr2lc\" (UniqueName: \"kubernetes.io/projected/60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac-kube-api-access-qr2lc\") pod \"60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac\" (UID: \"60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac\") " Nov 01 07:00:18.707234 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.707211 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b1e9b1-6c86-4818-bab1-715cad35fb8c-maas-api-service-ca" pod "f3b1e9b1-6c86-4818-bab1-715cad35fb8c" (UID: "f3b1e9b1-6c86-4818-bab1-715cad35fb8c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:00:18.710451 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.707481 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac-maas-api-service-ca" pod "60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac" (UID: "60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:00:18.710451 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.708553 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac-maas-api-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:00:18.710451 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.708578 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b1e9b1-6c86-4818-bab1-715cad35fb8c-maas-api-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:00:18.716564 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.716536 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv"] Nov 01 07:00:18.770726 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.770697 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b1e9b1-6c86-4818-bab1-715cad35fb8c-kube-api-access-xwffc" pod "f3b1e9b1-6c86-4818-bab1-715cad35fb8c" (UID: "f3b1e9b1-6c86-4818-bab1-715cad35fb8c"). InnerVolumeSpecName "kube-api-access-xwffc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:00:18.770889 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.770739 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac-kube-api-access-qr2lc" pod "60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac" (UID: "60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac"). InnerVolumeSpecName "kube-api-access-qr2lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:00:18.810241 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.810211 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr2lc\" (UniqueName: \"kubernetes.io/projected/60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac-kube-api-access-qr2lc\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:00:18.810241 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:18.810237 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwffc\" (UniqueName: \"kubernetes.io/projected/f3b1e9b1-6c86-4818-bab1-715cad35fb8c-kube-api-access-xwffc\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:00:19.413725 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.413440 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d" event={"ID":"60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac","Type":"ContainerDied","Data":"d1e878c11e3ab175e6a080270fd2a0eac69d6a86629bb90a0d12366830892877"} Nov 01 07:00:19.413725 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.413499 4385 scope.go:122] "RemoveContainer" containerID="83a54e45197f160ca7011bb9391b3652bfe614435bf0eba9b7a8f8dd3726442a" Nov 01 07:00:19.413725 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.413447 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d" Nov 01 07:00:19.417118 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.417091 4385 generic.go:356] "Generic (PLEG): container finished" podID="9debfa95-4fbc-4962-8177-6a53821135ca" containerID="4e02550c512b8435f174f7a6836638914be82c742d278f96b08f52f47d12654b" exitCode=60 Nov 01 07:00:19.418239 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.417167 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" event={"ID":"9debfa95-4fbc-4962-8177-6a53821135ca","Type":"ContainerDied","Data":"4e02550c512b8435f174f7a6836638914be82c742d278f96b08f52f47d12654b"} Nov 01 07:00:19.418239 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.417529 4385 scope.go:122] "RemoveContainer" containerID="4e02550c512b8435f174f7a6836638914be82c742d278f96b08f52f47d12654b" Nov 01 07:00:19.418239 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:19.417783 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp_odh-ai-gateway-infra(9debfa95-4fbc-4962-8177-6a53821135ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" podUID="9debfa95-4fbc-4962-8177-6a53821135ca" Nov 01 07:00:19.419560 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.419534 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" event={"ID":"247e6488-ee32-4931-9670-ea759c4cc896","Type":"ContainerStarted","Data":"117cba679b416dfbb5923a01adad3113f3866bfe22e1634e413cda5cfb022857"} Nov 01 07:00:19.423265 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.423245 4385 generic.go:356] "Generic (PLEG): container finished" podID="d27e5671-8155-4605-9333-001d14195f62" containerID="f7782e866517460d9715e58ed19fa4e863a5b73dd0532f0f2a772a7e0e62a847" exitCode=60 Nov 01 07:00:19.423382 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.423314 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" event={"ID":"d27e5671-8155-4605-9333-001d14195f62","Type":"ContainerDied","Data":"f7782e866517460d9715e58ed19fa4e863a5b73dd0532f0f2a772a7e0e62a847"} Nov 01 07:00:19.423671 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.423655 4385 scope.go:122] "RemoveContainer" containerID="f7782e866517460d9715e58ed19fa4e863a5b73dd0532f0f2a772a7e0e62a847" Nov 01 07:00:19.423998 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:19.423975 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk_odh-ai-gateway-infra(d27e5671-8155-4605-9333-001d14195f62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" podUID="d27e5671-8155-4605-9333-001d14195f62" Nov 01 07:00:19.424663 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.424645 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" event={"ID":"f3b1e9b1-6c86-4818-bab1-715cad35fb8c","Type":"ContainerDied","Data":"ed0ef98723ab753c42a8455872c9ee57c4f7e62134a5ae7c20a3105c81efdc8e"} Nov 01 07:00:19.424663 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.424654 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m" Nov 01 07:00:19.425257 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.425239 4385 scope.go:122] "RemoveContainer" containerID="634f0f1ffcf8b9d50513576cda329fcdfe1a1d54081be5c05a7f758b4c5b7bd0" Nov 01 07:00:19.431808 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.431786 4385 generic.go:356] "Generic (PLEG): container finished" podID="becaf42b-35ef-46db-a4f8-0cc8780722f0" containerID="654128d2b576ef11d923adc82c940ec8f8ed63ee51e04c6c0efc510114ce4bb8" exitCode=60 Nov 01 07:00:19.431984 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.431863 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" event={"ID":"becaf42b-35ef-46db-a4f8-0cc8780722f0","Type":"ContainerDied","Data":"654128d2b576ef11d923adc82c940ec8f8ed63ee51e04c6c0efc510114ce4bb8"} Nov 01 07:00:19.432208 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.432186 4385 scope.go:122] "RemoveContainer" containerID="654128d2b576ef11d923adc82c940ec8f8ed63ee51e04c6c0efc510114ce4bb8" Nov 01 07:00:19.432457 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:19.432435 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg_odh-ai-gateway-infra(becaf42b-35ef-46db-a4f8-0cc8780722f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" podUID="becaf42b-35ef-46db-a4f8-0cc8780722f0" Nov 01 07:00:19.441480 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.441458 4385 scope.go:122] "RemoveContainer" containerID="a777718b579604ddc4995191aada303f492a6093ecf8fe75474528249f0e4d38" Nov 01 07:00:19.456521 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.456498 4385 scope.go:122] "RemoveContainer" containerID="f6647f88146d302121072708417e79086dd6d3d5ee677dd2207422c6b245b153" Nov 01 07:00:19.467092 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.467005 4385 scope.go:122] "RemoveContainer" containerID="b0aaf99d71641997eaea03992a6982cec1c5737224fa17a47225da4206912fa4" Nov 01 07:00:19.640051 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.640029 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" Nov 01 07:00:19.722709 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.722635 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3871d20b-6d9f-4c6e-87b1-8e66cc4ef350-kube-api-access-6rxn6\" (UniqueName: \"kubernetes.io/projected/3871d20b-6d9f-4c6e-87b1-8e66cc4ef350-kube-api-access-6rxn6\") pod \"3871d20b-6d9f-4c6e-87b1-8e66cc4ef350\" (UID: \"3871d20b-6d9f-4c6e-87b1-8e66cc4ef350\") " Nov 01 07:00:19.722893 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.722760 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3871d20b-6d9f-4c6e-87b1-8e66cc4ef350-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3871d20b-6d9f-4c6e-87b1-8e66cc4ef350-maas-api-service-ca\") pod \"3871d20b-6d9f-4c6e-87b1-8e66cc4ef350\" (UID: \"3871d20b-6d9f-4c6e-87b1-8e66cc4ef350\") " Nov 01 07:00:19.723122 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.723093 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3871d20b-6d9f-4c6e-87b1-8e66cc4ef350-maas-api-service-ca" pod "3871d20b-6d9f-4c6e-87b1-8e66cc4ef350" (UID: "3871d20b-6d9f-4c6e-87b1-8e66cc4ef350"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:00:19.724891 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.724871 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3871d20b-6d9f-4c6e-87b1-8e66cc4ef350-kube-api-access-6rxn6" pod "3871d20b-6d9f-4c6e-87b1-8e66cc4ef350" (UID: "3871d20b-6d9f-4c6e-87b1-8e66cc4ef350"). InnerVolumeSpecName "kube-api-access-6rxn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:00:19.731568 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.731543 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg"] Nov 01 07:00:19.824150 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.824117 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3871d20b-6d9f-4c6e-87b1-8e66cc4ef350-maas-api-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:00:19.824150 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.824144 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rxn6\" (UniqueName: \"kubernetes.io/projected/3871d20b-6d9f-4c6e-87b1-8e66cc4ef350-kube-api-access-6rxn6\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:00:19.841558 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.841528 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk"] Nov 01 07:00:19.925721 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.925693 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d"] Nov 01 07:00:19.933050 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.933025 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dc6e48-29891940-tv78d"] Nov 01 07:00:19.970626 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:19.970593 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:20.042982 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.042897 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m"] Nov 01 07:00:20.044940 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.044920 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-caf74d-29891940-cmx4m"] Nov 01 07:00:20.065269 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.065246 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp"] Nov 01 07:00:20.390143 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.390111 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac" path="/var/lib/kubelet/pods/60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac/volumes" Nov 01 07:00:20.390819 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.390788 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b1e9b1-6c86-4818-bab1-715cad35fb8c" path="/var/lib/kubelet/pods/f3b1e9b1-6c86-4818-bab1-715cad35fb8c/volumes" Nov 01 07:00:20.444773 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.444747 4385 generic.go:356] "Generic (PLEG): container finished" podID="247e6488-ee32-4931-9670-ea759c4cc896" containerID="117cba679b416dfbb5923a01adad3113f3866bfe22e1634e413cda5cfb022857" exitCode=60 Nov 01 07:00:20.445119 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.444823 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" event={"ID":"247e6488-ee32-4931-9670-ea759c4cc896","Type":"ContainerDied","Data":"117cba679b416dfbb5923a01adad3113f3866bfe22e1634e413cda5cfb022857"} Nov 01 07:00:20.445119 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.444919 4385 scope.go:122] "RemoveContainer" containerID="506f848e4427f485d62b4b44932dd6397dc53df1e0d121df3c371ca7e489f137" Nov 01 07:00:20.445285 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.445268 4385 scope.go:122] "RemoveContainer" containerID="117cba679b416dfbb5923a01adad3113f3866bfe22e1634e413cda5cfb022857" Nov 01 07:00:20.445549 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:20.445527 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s_odh-ai-gateway-infra(247e6488-ee32-4931-9670-ea759c4cc896)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" podUID="247e6488-ee32-4931-9670-ea759c4cc896" Nov 01 07:00:20.451121 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.451091 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" event={"ID":"3871d20b-6d9f-4c6e-87b1-8e66cc4ef350","Type":"ContainerDied","Data":"852aed23260330c6e4deafc6cee554f0ba0e6df83eeb2c2bc75415a72960f3f4"} Nov 01 07:00:20.451226 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.451108 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv" Nov 01 07:00:20.460896 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.460859 4385 scope.go:122] "RemoveContainer" containerID="9f4ff76eb3e87a917f2d24b37a6db7c9030407b6988f1f6e7bf4100d537ece5b" Nov 01 07:00:20.593692 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.593567 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s"] Nov 01 07:00:20.638676 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.638637 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" Nov 01 07:00:20.676124 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.676101 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" Nov 01 07:00:20.714435 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.714414 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" Nov 01 07:00:20.735175 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.735155 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d27e5671-8155-4605-9333-001d14195f62-kube-api-access-ktgl7\" (UniqueName: \"kubernetes.io/projected/d27e5671-8155-4605-9333-001d14195f62-kube-api-access-ktgl7\") pod \"d27e5671-8155-4605-9333-001d14195f62\" (UID: \"d27e5671-8155-4605-9333-001d14195f62\") " Nov 01 07:00:20.735323 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.735215 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d27e5671-8155-4605-9333-001d14195f62-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d27e5671-8155-4605-9333-001d14195f62-maas-api-service-ca\") pod \"d27e5671-8155-4605-9333-001d14195f62\" (UID: \"d27e5671-8155-4605-9333-001d14195f62\") " Nov 01 07:00:20.735594 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.735572 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27e5671-8155-4605-9333-001d14195f62-maas-api-service-ca" pod "d27e5671-8155-4605-9333-001d14195f62" (UID: "d27e5671-8155-4605-9333-001d14195f62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:00:20.735799 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.735779 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d27e5671-8155-4605-9333-001d14195f62-maas-api-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:00:20.737472 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.737442 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27e5671-8155-4605-9333-001d14195f62-kube-api-access-ktgl7" pod "d27e5671-8155-4605-9333-001d14195f62" (UID: "d27e5671-8155-4605-9333-001d14195f62"). InnerVolumeSpecName "kube-api-access-ktgl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:00:20.782303 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.782275 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv"] Nov 01 07:00:20.792291 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.792257 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c79abe-29891940-sgcpv"] Nov 01 07:00:20.836476 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.836450 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9debfa95-4fbc-4962-8177-6a53821135ca-kube-api-access-6nhvm\" (UniqueName: \"kubernetes.io/projected/9debfa95-4fbc-4962-8177-6a53821135ca-kube-api-access-6nhvm\") pod \"9debfa95-4fbc-4962-8177-6a53821135ca\" (UID: \"9debfa95-4fbc-4962-8177-6a53821135ca\") " Nov 01 07:00:20.836632 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.836491 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/becaf42b-35ef-46db-a4f8-0cc8780722f0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/becaf42b-35ef-46db-a4f8-0cc8780722f0-maas-api-service-ca\") pod \"becaf42b-35ef-46db-a4f8-0cc8780722f0\" (UID: \"becaf42b-35ef-46db-a4f8-0cc8780722f0\") " Nov 01 07:00:20.836632 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.836557 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9debfa95-4fbc-4962-8177-6a53821135ca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9debfa95-4fbc-4962-8177-6a53821135ca-maas-api-service-ca\") pod \"9debfa95-4fbc-4962-8177-6a53821135ca\" (UID: \"9debfa95-4fbc-4962-8177-6a53821135ca\") " Nov 01 07:00:20.836763 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.836698 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/becaf42b-35ef-46db-a4f8-0cc8780722f0-kube-api-access-pgd7r\" (UniqueName: \"kubernetes.io/projected/becaf42b-35ef-46db-a4f8-0cc8780722f0-kube-api-access-pgd7r\") pod \"becaf42b-35ef-46db-a4f8-0cc8780722f0\" (UID: \"becaf42b-35ef-46db-a4f8-0cc8780722f0\") " Nov 01 07:00:20.837007 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.836978 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9debfa95-4fbc-4962-8177-6a53821135ca-maas-api-service-ca" pod "9debfa95-4fbc-4962-8177-6a53821135ca" (UID: "9debfa95-4fbc-4962-8177-6a53821135ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:00:20.837132 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.836977 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/becaf42b-35ef-46db-a4f8-0cc8780722f0-maas-api-service-ca" pod "becaf42b-35ef-46db-a4f8-0cc8780722f0" (UID: "becaf42b-35ef-46db-a4f8-0cc8780722f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:00:20.837206 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.837192 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/becaf42b-35ef-46db-a4f8-0cc8780722f0-maas-api-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:00:20.837260 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.837213 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9debfa95-4fbc-4962-8177-6a53821135ca-maas-api-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:00:20.837260 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.837228 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktgl7\" (UniqueName: \"kubernetes.io/projected/d27e5671-8155-4605-9333-001d14195f62-kube-api-access-ktgl7\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:00:20.838784 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.838765 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9debfa95-4fbc-4962-8177-6a53821135ca-kube-api-access-6nhvm" pod "9debfa95-4fbc-4962-8177-6a53821135ca" (UID: "9debfa95-4fbc-4962-8177-6a53821135ca"). InnerVolumeSpecName "kube-api-access-6nhvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:00:20.838870 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.838817 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/becaf42b-35ef-46db-a4f8-0cc8780722f0-kube-api-access-pgd7r" pod "becaf42b-35ef-46db-a4f8-0cc8780722f0" (UID: "becaf42b-35ef-46db-a4f8-0cc8780722f0"). InnerVolumeSpecName "kube-api-access-pgd7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:00:20.938502 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.938431 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgd7r\" (UniqueName: \"kubernetes.io/projected/becaf42b-35ef-46db-a4f8-0cc8780722f0-kube-api-access-pgd7r\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:00:20.938502 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:20.938456 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nhvm\" (UniqueName: \"kubernetes.io/projected/9debfa95-4fbc-4962-8177-6a53821135ca-kube-api-access-6nhvm\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:00:21.464740 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.464716 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" Nov 01 07:00:21.465190 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.464740 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk" event={"ID":"d27e5671-8155-4605-9333-001d14195f62","Type":"ContainerDied","Data":"f06b1dd07b16f8570433fe42510df2cbc650b9402b00dfba8e3a21d4fdf52edf"} Nov 01 07:00:21.465190 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.464805 4385 scope.go:122] "RemoveContainer" containerID="f7782e866517460d9715e58ed19fa4e863a5b73dd0532f0f2a772a7e0e62a847" Nov 01 07:00:21.467846 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.467811 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" Nov 01 07:00:21.467953 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.467870 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg" event={"ID":"becaf42b-35ef-46db-a4f8-0cc8780722f0","Type":"ContainerDied","Data":"b1b496fb0860dbe95c09b0f6decb6f86905fd67d5d1acb6a68812b0cfb145993"} Nov 01 07:00:21.469823 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.469801 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" Nov 01 07:00:21.469823 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.469808 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp" event={"ID":"9debfa95-4fbc-4962-8177-6a53821135ca","Type":"ContainerDied","Data":"0de042826c21597ea82aa1f67837c3d6821783249a25547abbd5e44d87f47d20"} Nov 01 07:00:21.481526 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.481504 4385 scope.go:122] "RemoveContainer" containerID="654128d2b576ef11d923adc82c940ec8f8ed63ee51e04c6c0efc510114ce4bb8" Nov 01 07:00:21.498013 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.497972 4385 scope.go:122] "RemoveContainer" containerID="4e02550c512b8435f174f7a6836638914be82c742d278f96b08f52f47d12654b" Nov 01 07:00:21.674038 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.674012 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" Nov 01 07:00:21.693574 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.693550 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg"] Nov 01 07:00:21.699343 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.699315 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6ba6e5-29891940-959gg"] Nov 01 07:00:21.805853 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.805749 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk"] Nov 01 07:00:21.809541 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.809517 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e0116-29891940-2czvk"] Nov 01 07:00:21.848938 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.848910 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/247e6488-ee32-4931-9670-ea759c4cc896-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247e6488-ee32-4931-9670-ea759c4cc896-maas-api-service-ca\") pod \"247e6488-ee32-4931-9670-ea759c4cc896\" (UID: \"247e6488-ee32-4931-9670-ea759c4cc896\") " Nov 01 07:00:21.849096 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.849062 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/247e6488-ee32-4931-9670-ea759c4cc896-kube-api-access-zz9jp\" (UniqueName: \"kubernetes.io/projected/247e6488-ee32-4931-9670-ea759c4cc896-kube-api-access-zz9jp\") pod \"247e6488-ee32-4931-9670-ea759c4cc896\" (UID: \"247e6488-ee32-4931-9670-ea759c4cc896\") " Nov 01 07:00:21.849242 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.849218 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247e6488-ee32-4931-9670-ea759c4cc896-maas-api-service-ca" pod "247e6488-ee32-4931-9670-ea759c4cc896" (UID: "247e6488-ee32-4931-9670-ea759c4cc896"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:00:21.849499 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.849479 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/247e6488-ee32-4931-9670-ea759c4cc896-maas-api-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:00:21.851238 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.851218 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247e6488-ee32-4931-9670-ea759c4cc896-kube-api-access-zz9jp" pod "247e6488-ee32-4931-9670-ea759c4cc896" (UID: "247e6488-ee32-4931-9670-ea759c4cc896"). InnerVolumeSpecName "kube-api-access-zz9jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:00:21.918828 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.918779 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp"] Nov 01 07:00:21.922871 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.922844 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-0cb4ef-29891940-gm7fp"] Nov 01 07:00:21.950873 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:21.950821 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz9jp\" (UniqueName: \"kubernetes.io/projected/247e6488-ee32-4931-9670-ea759c4cc896-kube-api-access-zz9jp\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:00:22.395754 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:22.395719 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3871d20b-6d9f-4c6e-87b1-8e66cc4ef350" path="/var/lib/kubelet/pods/3871d20b-6d9f-4c6e-87b1-8e66cc4ef350/volumes" Nov 01 07:00:22.396347 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:22.396329 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9debfa95-4fbc-4962-8177-6a53821135ca" path="/var/lib/kubelet/pods/9debfa95-4fbc-4962-8177-6a53821135ca/volumes" Nov 01 07:00:22.396853 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:22.396821 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="becaf42b-35ef-46db-a4f8-0cc8780722f0" path="/var/lib/kubelet/pods/becaf42b-35ef-46db-a4f8-0cc8780722f0/volumes" Nov 01 07:00:22.397751 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:22.397734 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d27e5671-8155-4605-9333-001d14195f62" path="/var/lib/kubelet/pods/d27e5671-8155-4605-9333-001d14195f62/volumes" Nov 01 07:00:22.482324 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:22.482287 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" event={"ID":"247e6488-ee32-4931-9670-ea759c4cc896","Type":"ContainerDied","Data":"89561fcef3b5ef13797fb437f5b5090870c235061149d8e3fed722a429732883"} Nov 01 07:00:22.482792 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:22.482359 4385 scope.go:122] "RemoveContainer" containerID="117cba679b416dfbb5923a01adad3113f3866bfe22e1634e413cda5cfb022857" Nov 01 07:00:22.482792 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:22.482452 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s" Nov 01 07:00:22.647646 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:22.647558 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s"] Nov 01 07:00:22.651132 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:22.651096 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8ff537-29891940-xk57s"] Nov 01 07:00:24.388990 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:24.388961 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="247e6488-ee32-4931-9670-ea759c4cc896" path="/var/lib/kubelet/pods/247e6488-ee32-4931-9670-ea759c4cc896/volumes" Nov 01 07:00:24.679761 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:24.679669 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:27.141754 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:27.141722 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:29.449602 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:29.449564 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:34.125148 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:34.125112 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:36.779882 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:00:36.779844 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:00:38.818024 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:38.817979 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:40.325238 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:40.325204 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:40.471845 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:40.471804 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:43.371242 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:43.371203 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:43.561903 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:43.561867 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:43.701153 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:43.701061 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:44.717942 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:44.717905 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:45.188800 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:45.188764 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:48.285029 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:48.284992 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:50.229268 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:50.229230 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:51.115460 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:51.115422 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:53.031480 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:53.031423 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:00:57.678130 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:00:57.678097 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:01:02.339584 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:02.339526 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:01:06.783514 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:01:06.783478 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:01:07.031243 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:07.031211 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:01:11.666553 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:11.666518 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:01:16.317155 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:16.317114 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:01:21.143244 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:21.143206 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:01:25.838797 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:25.838761 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:01:27.185282 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:27.185244 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:01:27.709769 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:27.709733 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj"] Nov 01 07:01:27.976463 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:27.976375 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:01:28.124303 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:28.124264 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:01:30.981919 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:30.981875 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:01:32.599050 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:32.599015 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:01:32.710331 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:32.710290 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj" podUID="22d2e118-2604-4c39-9f1a-cf6a509397fa" containerName="payload-pre-processing" containerID="cri-o://d37fb46e317d9677017fbad1ea6eabbba0e8b7af5f017e8fbea33e4ea9456e9c" gracePeriod=25 Nov 01 07:01:33.007751 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:33.007718 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj" Nov 01 07:01:33.039042 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:33.039005 4385 generic.go:356] "Generic (PLEG): container finished" podID="22d2e118-2604-4c39-9f1a-cf6a509397fa" containerID="d37fb46e317d9677017fbad1ea6eabbba0e8b7af5f017e8fbea33e4ea9456e9c" exitCode=0 Nov 01 07:01:33.039216 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:33.039061 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj" Nov 01 07:01:33.039216 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:33.039090 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj" event={"ID":"22d2e118-2604-4c39-9f1a-cf6a509397fa","Type":"ContainerDied","Data":"d37fb46e317d9677017fbad1ea6eabbba0e8b7af5f017e8fbea33e4ea9456e9c"} Nov 01 07:01:33.039216 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:33.039149 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj" event={"ID":"22d2e118-2604-4c39-9f1a-cf6a509397fa","Type":"ContainerDied","Data":"c41b0623271572ac6db1ec42b6d833fa6f0a8d4581fb6d677bfc97c7395f4ee1"} Nov 01 07:01:33.039216 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:33.039180 4385 scope.go:122] "RemoveContainer" containerID="d37fb46e317d9677017fbad1ea6eabbba0e8b7af5f017e8fbea33e4ea9456e9c" Nov 01 07:01:33.050122 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:33.050101 4385 scope.go:122] "RemoveContainer" containerID="d37fb46e317d9677017fbad1ea6eabbba0e8b7af5f017e8fbea33e4ea9456e9c" Nov 01 07:01:33.050512 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:01:33.050475 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37fb46e317d9677017fbad1ea6eabbba0e8b7af5f017e8fbea33e4ea9456e9c\": container with ID starting with d37fb46e317d9677017fbad1ea6eabbba0e8b7af5f017e8fbea33e4ea9456e9c not found: ID does not exist" containerID="d37fb46e317d9677017fbad1ea6eabbba0e8b7af5f017e8fbea33e4ea9456e9c" Nov 01 07:01:33.050565 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:33.050508 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37fb46e317d9677017fbad1ea6eabbba0e8b7af5f017e8fbea33e4ea9456e9c"} err="failed to get container status \"d37fb46e317d9677017fbad1ea6eabbba0e8b7af5f017e8fbea33e4ea9456e9c\": rpc error: code = NotFound desc = could not find container \"d37fb46e317d9677017fbad1ea6eabbba0e8b7af5f017e8fbea33e4ea9456e9c\": container with ID starting with d37fb46e317d9677017fbad1ea6eabbba0e8b7af5f017e8fbea33e4ea9456e9c not found: ID does not exist" Nov 01 07:01:33.167689 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:33.167609 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22d2e118-2604-4c39-9f1a-cf6a509397fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d2e118-2604-4c39-9f1a-cf6a509397fa-plugins-config-volume\") pod \"22d2e118-2604-4c39-9f1a-cf6a509397fa\" (UID: \"22d2e118-2604-4c39-9f1a-cf6a509397fa\") " Nov 01 07:01:33.167895 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:33.167694 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22d2e118-2604-4c39-9f1a-cf6a509397fa-kube-api-access-c8vvb\" (UniqueName: \"kubernetes.io/projected/22d2e118-2604-4c39-9f1a-cf6a509397fa-kube-api-access-c8vvb\") pod \"22d2e118-2604-4c39-9f1a-cf6a509397fa\" (UID: \"22d2e118-2604-4c39-9f1a-cf6a509397fa\") " Nov 01 07:01:33.168076 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:33.168044 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d2e118-2604-4c39-9f1a-cf6a509397fa-plugins-config-volume" pod "22d2e118-2604-4c39-9f1a-cf6a509397fa" (UID: "22d2e118-2604-4c39-9f1a-cf6a509397fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:01:33.170148 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:33.170127 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d2e118-2604-4c39-9f1a-cf6a509397fa-kube-api-access-c8vvb" pod "22d2e118-2604-4c39-9f1a-cf6a509397fa" (UID: "22d2e118-2604-4c39-9f1a-cf6a509397fa"). InnerVolumeSpecName "kube-api-access-c8vvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:01:33.269672 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:33.269637 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8vvb\" (UniqueName: \"kubernetes.io/projected/22d2e118-2604-4c39-9f1a-cf6a509397fa-kube-api-access-c8vvb\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:01:33.269672 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:33.269670 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d2e118-2604-4c39-9f1a-cf6a509397fa-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:01:33.462353 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:33.462263 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj"] Nov 01 07:01:33.465910 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:33.465886 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0cb4ef-cd664b6f9-5wvkj"] Nov 01 07:01:34.387130 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:34.387093 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22d2e118-2604-4c39-9f1a-cf6a509397fa" path="/var/lib/kubelet/pods/22d2e118-2604-4c39-9f1a-cf6a509397fa/volumes" Nov 01 07:01:36.231516 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:36.231471 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:01:36.782910 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:01:36.782862 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:01:41.259590 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:41.259555 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:01:42.619878 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:42.619821 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n"] Nov 01 07:01:42.990687 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:42.990589 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:01:43.129467 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:43.129421 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:01:45.961895 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:45.961855 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:01:47.620568 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:47.620515 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n" podUID="250ce347-5c48-4490-b942-0f4916c707a5" containerName="payload-processing" containerID="cri-o://4b481459890a78fc9ffa780bacef97b5d90981e85d00f01696fc67ab1aabce67" gracePeriod=25 Nov 01 07:01:50.620713 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:50.620675 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:01:51.453574 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:51.453538 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:01:52.696417 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:52.696385 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n" podUID="250ce347-5c48-4490-b942-0f4916c707a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:01:55.314659 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:55.314618 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:01:59.982365 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:01:59.982314 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:02.696786 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:02.696744 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n" podUID="250ce347-5c48-4490-b942-0f4916c707a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:02:04.723099 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:04.723056 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:06.782499 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:02:06.782460 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:02:07.921442 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:07.921412 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n" Nov 01 07:02:08.007812 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:08.007778 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/250ce347-5c48-4490-b942-0f4916c707a5-kube-api-access-cgxxn\" (UniqueName: \"kubernetes.io/projected/250ce347-5c48-4490-b942-0f4916c707a5-kube-api-access-cgxxn\") pod \"250ce347-5c48-4490-b942-0f4916c707a5\" (UID: \"250ce347-5c48-4490-b942-0f4916c707a5\") " Nov 01 07:02:08.008031 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:08.007916 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/250ce347-5c48-4490-b942-0f4916c707a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250ce347-5c48-4490-b942-0f4916c707a5-plugins-config-volume\") pod \"250ce347-5c48-4490-b942-0f4916c707a5\" (UID: \"250ce347-5c48-4490-b942-0f4916c707a5\") " Nov 01 07:02:08.008316 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:08.008287 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250ce347-5c48-4490-b942-0f4916c707a5-plugins-config-volume" pod "250ce347-5c48-4490-b942-0f4916c707a5" (UID: "250ce347-5c48-4490-b942-0f4916c707a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:02:08.008447 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:08.008430 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250ce347-5c48-4490-b942-0f4916c707a5-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:08.010248 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:08.010220 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250ce347-5c48-4490-b942-0f4916c707a5-kube-api-access-cgxxn" pod "250ce347-5c48-4490-b942-0f4916c707a5" (UID: "250ce347-5c48-4490-b942-0f4916c707a5"). InnerVolumeSpecName "kube-api-access-cgxxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:02:08.109769 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:08.109735 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgxxn\" (UniqueName: \"kubernetes.io/projected/250ce347-5c48-4490-b942-0f4916c707a5-kube-api-access-cgxxn\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:08.304075 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:08.303978 4385 generic.go:356] "Generic (PLEG): container finished" podID="250ce347-5c48-4490-b942-0f4916c707a5" containerID="4b481459890a78fc9ffa780bacef97b5d90981e85d00f01696fc67ab1aabce67" exitCode=0 Nov 01 07:02:08.304075 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:08.304023 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n" Nov 01 07:02:08.304075 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:08.304052 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n" event={"ID":"250ce347-5c48-4490-b942-0f4916c707a5","Type":"ContainerDied","Data":"4b481459890a78fc9ffa780bacef97b5d90981e85d00f01696fc67ab1aabce67"} Nov 01 07:02:08.304333 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:08.304105 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n" event={"ID":"250ce347-5c48-4490-b942-0f4916c707a5","Type":"ContainerDied","Data":"4361fef4ee46f6c4def71186087d51100fcb510ae50971b03b1c3ce513449f41"} Nov 01 07:02:08.304333 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:08.304134 4385 scope.go:122] "RemoveContainer" containerID="4b481459890a78fc9ffa780bacef97b5d90981e85d00f01696fc67ab1aabce67" Nov 01 07:02:08.321955 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:08.321934 4385 scope.go:122] "RemoveContainer" containerID="4b481459890a78fc9ffa780bacef97b5d90981e85d00f01696fc67ab1aabce67" Nov 01 07:02:08.322327 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:02:08.322305 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b481459890a78fc9ffa780bacef97b5d90981e85d00f01696fc67ab1aabce67\": container with ID starting with 4b481459890a78fc9ffa780bacef97b5d90981e85d00f01696fc67ab1aabce67 not found: ID does not exist" containerID="4b481459890a78fc9ffa780bacef97b5d90981e85d00f01696fc67ab1aabce67" Nov 01 07:02:08.322420 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:08.322340 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b481459890a78fc9ffa780bacef97b5d90981e85d00f01696fc67ab1aabce67"} err="failed to get container status \"4b481459890a78fc9ffa780bacef97b5d90981e85d00f01696fc67ab1aabce67\": rpc error: code = NotFound desc = could not find container \"4b481459890a78fc9ffa780bacef97b5d90981e85d00f01696fc67ab1aabce67\": container with ID starting with 4b481459890a78fc9ffa780bacef97b5d90981e85d00f01696fc67ab1aabce67 not found: ID does not exist" Nov 01 07:02:08.467768 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:08.467733 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n"] Nov 01 07:02:08.517161 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:08.517122 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c831a4c0-6fdb86cf8f-5wd2n"] Nov 01 07:02:09.425526 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:09.425488 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:10.388134 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:10.388099 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="250ce347-5c48-4490-b942-0f4916c707a5" path="/var/lib/kubelet/pods/250ce347-5c48-4490-b942-0f4916c707a5/volumes" Nov 01 07:02:14.107733 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:14.107696 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:15.412293 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:15.412254 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w"] Nov 01 07:02:15.412763 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:15.412542 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" podUID="25f4c33e-6491-4057-8979-f8ff03d56c6f" containerName="istio-proxy" containerID="cri-o://16a61ffe1da8eb241d86ff3014592dae5a02633ae16c21984275132ab187fd3c" gracePeriod=30 Nov 01 07:02:18.962050 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:18.962007 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-njl8w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 07:02:18.962460 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:18.962077 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" podUID="25f4c33e-6491-4057-8979-f8ff03d56c6f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 07:02:19.161624 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:19.161586 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:20.703694 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.703630 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 07:02:20.758192 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.758163 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-workload-socket\") pod \"25f4c33e-6491-4057-8979-f8ff03d56c6f\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " Nov 01 07:02:20.758381 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.758244 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25f4c33e-6491-4057-8979-f8ff03d56c6f-kube-api-access-79qj4\" (UniqueName: \"kubernetes.io/projected/25f4c33e-6491-4057-8979-f8ff03d56c6f-kube-api-access-79qj4\") pod \"25f4c33e-6491-4057-8979-f8ff03d56c6f\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " Nov 01 07:02:20.758381 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.758303 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25f4c33e-6491-4057-8979-f8ff03d56c6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25f4c33e-6491-4057-8979-f8ff03d56c6f-istiod-ca-cert\") pod \"25f4c33e-6491-4057-8979-f8ff03d56c6f\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " Nov 01 07:02:20.758381 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.758335 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-credential-socket\") pod \"25f4c33e-6491-4057-8979-f8ff03d56c6f\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " Nov 01 07:02:20.758381 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.758360 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-token\" (UniqueName: \"kubernetes.io/projected/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-token\") pod \"25f4c33e-6491-4057-8979-f8ff03d56c6f\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " Nov 01 07:02:20.758381 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.758381 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-data\") pod \"25f4c33e-6491-4057-8979-f8ff03d56c6f\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " Nov 01 07:02:20.758640 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.758413 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-workload-certs\") pod \"25f4c33e-6491-4057-8979-f8ff03d56c6f\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " Nov 01 07:02:20.758640 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.758475 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-envoy\") pod \"25f4c33e-6491-4057-8979-f8ff03d56c6f\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " Nov 01 07:02:20.758640 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.758490 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-workload-socket" pod "25f4c33e-6491-4057-8979-f8ff03d56c6f" (UID: "25f4c33e-6491-4057-8979-f8ff03d56c6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:02:20.758640 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.758500 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-podinfo\") pod \"25f4c33e-6491-4057-8979-f8ff03d56c6f\" (UID: \"25f4c33e-6491-4057-8979-f8ff03d56c6f\") " Nov 01 07:02:20.758895 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.758677 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-credential-socket" pod "25f4c33e-6491-4057-8979-f8ff03d56c6f" (UID: "25f4c33e-6491-4057-8979-f8ff03d56c6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:02:20.758895 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.758709 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-data" pod "25f4c33e-6491-4057-8979-f8ff03d56c6f" (UID: "25f4c33e-6491-4057-8979-f8ff03d56c6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:02:20.758895 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.758777 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25f4c33e-6491-4057-8979-f8ff03d56c6f-istiod-ca-cert" pod "25f4c33e-6491-4057-8979-f8ff03d56c6f" (UID: "25f4c33e-6491-4057-8979-f8ff03d56c6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:02:20.758895 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.758872 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-workload-certs" pod "25f4c33e-6491-4057-8979-f8ff03d56c6f" (UID: "25f4c33e-6491-4057-8979-f8ff03d56c6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:02:20.759304 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.759225 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:20.759304 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.759259 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25f4c33e-6491-4057-8979-f8ff03d56c6f-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:20.759304 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.759270 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:20.759304 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.759279 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:20.759304 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.759288 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:20.760845 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.760807 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f4c33e-6491-4057-8979-f8ff03d56c6f-kube-api-access-79qj4" pod "25f4c33e-6491-4057-8979-f8ff03d56c6f" (UID: "25f4c33e-6491-4057-8979-f8ff03d56c6f"). InnerVolumeSpecName "kube-api-access-79qj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:02:20.760945 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.760882 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-podinfo" pod "25f4c33e-6491-4057-8979-f8ff03d56c6f" (UID: "25f4c33e-6491-4057-8979-f8ff03d56c6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:02:20.761261 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.761233 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-token" pod "25f4c33e-6491-4057-8979-f8ff03d56c6f" (UID: "25f4c33e-6491-4057-8979-f8ff03d56c6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:02:20.761337 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.761270 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-envoy" pod "25f4c33e-6491-4057-8979-f8ff03d56c6f" (UID: "25f4c33e-6491-4057-8979-f8ff03d56c6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:02:20.860264 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.860230 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:20.860264 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.860261 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:20.860490 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.860271 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25f4c33e-6491-4057-8979-f8ff03d56c6f-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:20.860490 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:20.860283 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79qj4\" (UniqueName: \"kubernetes.io/projected/25f4c33e-6491-4057-8979-f8ff03d56c6f-kube-api-access-79qj4\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:21.435940 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:21.435908 4385 generic.go:356] "Generic (PLEG): container finished" podID="25f4c33e-6491-4057-8979-f8ff03d56c6f" containerID="16a61ffe1da8eb241d86ff3014592dae5a02633ae16c21984275132ab187fd3c" exitCode=0 Nov 01 07:02:21.436117 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:21.435952 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" Nov 01 07:02:21.436117 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:21.435994 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" event={"ID":"25f4c33e-6491-4057-8979-f8ff03d56c6f","Type":"ContainerDied","Data":"16a61ffe1da8eb241d86ff3014592dae5a02633ae16c21984275132ab187fd3c"} Nov 01 07:02:21.436117 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:21.436052 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w" event={"ID":"25f4c33e-6491-4057-8979-f8ff03d56c6f","Type":"ContainerDied","Data":"95c7a9186dfd807776aab1d588f37af9333ea6474e1240d2240e7365bee93d2b"} Nov 01 07:02:21.436117 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:21.436094 4385 scope.go:122] "RemoveContainer" containerID="16a61ffe1da8eb241d86ff3014592dae5a02633ae16c21984275132ab187fd3c" Nov 01 07:02:21.447997 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:21.447981 4385 scope.go:122] "RemoveContainer" containerID="16a61ffe1da8eb241d86ff3014592dae5a02633ae16c21984275132ab187fd3c" Nov 01 07:02:21.448274 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:02:21.448256 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16a61ffe1da8eb241d86ff3014592dae5a02633ae16c21984275132ab187fd3c\": container with ID starting with 16a61ffe1da8eb241d86ff3014592dae5a02633ae16c21984275132ab187fd3c not found: ID does not exist" containerID="16a61ffe1da8eb241d86ff3014592dae5a02633ae16c21984275132ab187fd3c" Nov 01 07:02:21.448372 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:21.448278 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16a61ffe1da8eb241d86ff3014592dae5a02633ae16c21984275132ab187fd3c"} err="failed to get container status \"16a61ffe1da8eb241d86ff3014592dae5a02633ae16c21984275132ab187fd3c\": rpc error: code = NotFound desc = could not find container \"16a61ffe1da8eb241d86ff3014592dae5a02633ae16c21984275132ab187fd3c\": container with ID starting with 16a61ffe1da8eb241d86ff3014592dae5a02633ae16c21984275132ab187fd3c not found: ID does not exist" Nov 01 07:02:21.605965 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:21.605931 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w"] Nov 01 07:02:21.611377 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:21.611352 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-njl8w"] Nov 01 07:02:21.738295 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:21.738205 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:22.384876 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:22.384826 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25f4c33e-6491-4057-8979-f8ff03d56c6f" path="/var/lib/kubelet/pods/25f4c33e-6491-4057-8979-f8ff03d56c6f/volumes" Nov 01 07:02:23.920165 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:23.920134 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:27.604821 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:27.604783 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:28.265899 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:28.265864 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn"] Nov 01 07:02:28.640534 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:28.640493 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:28.836649 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:28.836611 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:33.104173 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:33.104133 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:33.266516 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:33.266477 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn" podUID="7d064de5-6ab4-45ab-a83a-0fe33a874788" containerName="payload-processing" containerID="cri-o://567fd33c3a411e072e2e0fa93bd813557c820661d7f7d0771f5c713eb658e74d" gracePeriod=25 Nov 01 07:02:33.525752 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:33.525717 4385 generic.go:356] "Generic (PLEG): container finished" podID="7d064de5-6ab4-45ab-a83a-0fe33a874788" containerID="567fd33c3a411e072e2e0fa93bd813557c820661d7f7d0771f5c713eb658e74d" exitCode=0 Nov 01 07:02:33.525940 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:33.525786 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn" event={"ID":"7d064de5-6ab4-45ab-a83a-0fe33a874788","Type":"ContainerDied","Data":"567fd33c3a411e072e2e0fa93bd813557c820661d7f7d0771f5c713eb658e74d"} Nov 01 07:02:33.525940 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:33.525880 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn" event={"ID":"7d064de5-6ab4-45ab-a83a-0fe33a874788","Type":"ContainerDied","Data":"faafcbfdac2b37995db72f5466408661672b618e7cd58f5f9d88dd722cb23acd"} Nov 01 07:02:33.525940 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:33.525907 4385 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faafcbfdac2b37995db72f5466408661672b618e7cd58f5f9d88dd722cb23acd" Nov 01 07:02:33.532453 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:33.532432 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn" Nov 01 07:02:33.619030 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:33.619004 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d064de5-6ab4-45ab-a83a-0fe33a874788-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d064de5-6ab4-45ab-a83a-0fe33a874788-plugins-config-volume\") pod \"7d064de5-6ab4-45ab-a83a-0fe33a874788\" (UID: \"7d064de5-6ab4-45ab-a83a-0fe33a874788\") " Nov 01 07:02:33.619214 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:33.619115 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d064de5-6ab4-45ab-a83a-0fe33a874788-kube-api-access-jvjv4\" (UniqueName: \"kubernetes.io/projected/7d064de5-6ab4-45ab-a83a-0fe33a874788-kube-api-access-jvjv4\") pod \"7d064de5-6ab4-45ab-a83a-0fe33a874788\" (UID: \"7d064de5-6ab4-45ab-a83a-0fe33a874788\") " Nov 01 07:02:33.619412 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:33.619389 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d064de5-6ab4-45ab-a83a-0fe33a874788-plugins-config-volume" pod "7d064de5-6ab4-45ab-a83a-0fe33a874788" (UID: "7d064de5-6ab4-45ab-a83a-0fe33a874788"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:02:33.619544 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:33.619528 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d064de5-6ab4-45ab-a83a-0fe33a874788-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:33.621293 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:33.621274 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d064de5-6ab4-45ab-a83a-0fe33a874788-kube-api-access-jvjv4" pod "7d064de5-6ab4-45ab-a83a-0fe33a874788" (UID: "7d064de5-6ab4-45ab-a83a-0fe33a874788"). InnerVolumeSpecName "kube-api-access-jvjv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:02:33.721192 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:33.721156 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvjv4\" (UniqueName: \"kubernetes.io/projected/7d064de5-6ab4-45ab-a83a-0fe33a874788-kube-api-access-jvjv4\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:33.722995 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:33.722969 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:34.532554 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:34.532523 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn" Nov 01 07:02:34.697213 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:34.697127 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn"] Nov 01 07:02:34.748618 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:34.748580 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-3db672-86f55c8c65-whcqn"] Nov 01 07:02:36.387237 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:36.387202 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d064de5-6ab4-45ab-a83a-0fe33a874788" path="/var/lib/kubelet/pods/7d064de5-6ab4-45ab-a83a-0fe33a874788/volumes" Nov 01 07:02:36.780870 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:02:36.780754 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:02:37.801532 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:37.801496 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:39.088523 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:39.088484 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:39.233323 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:39.233286 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:39.528936 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:39.528814 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:39.675576 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:39.675543 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:40.083612 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:40.083572 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:40.693572 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:40.693542 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:41.290620 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:41.290584 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:41.513915 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:41.513881 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:43.135058 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:43.135021 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:43.340511 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:43.340475 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:43.501008 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:43.500916 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:44.442306 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:44.442268 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:46.194693 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:46.194651 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:49.155603 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:49.155565 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:51.803076 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:51.803047 4385 scope.go:122] "RemoveContainer" containerID="567fd33c3a411e072e2e0fa93bd813557c820661d7f7d0771f5c713eb658e74d" Nov 01 07:02:53.167026 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:53.166989 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:53.834592 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:53.834543 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:02:58.530050 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:02:58.530007 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:03.243014 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:03.242925 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:06.780951 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:03:06.780913 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:03:07.855188 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:07.855149 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:12.507121 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:12.507050 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:13.413342 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:13.413303 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:13.614946 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:13.614914 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:13.892554 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:13.892514 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:14.042350 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:14.042308 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:17.234874 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:17.234809 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:19.068605 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:19.068565 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:21.376602 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:21.376565 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:22.805412 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:22.805374 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:23.002875 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:23.002825 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:23.151011 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:23.150974 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:24.401488 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:24.401454 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:24.862994 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:24.862954 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9"] Nov 01 07:03:24.863227 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:24.863196 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" podUID="5efe5a5b-a74f-4f75-a733-619770f4790c" containerName="istio-proxy" containerID="cri-o://48860f04357f0c4a9a02c116f62113329a6e3163606529d247979b9467b61841" gracePeriod=30 Nov 01 07:03:24.885539 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:24.885504 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:25.629417 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:25.629384 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:27.750175 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:27.750141 4385 patch_prober.go:29] interesting pod/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 07:03:27.750592 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:27.750202 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" podUID="5efe5a5b-a74f-4f75-a733-619770f4790c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 07:03:28.047586 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:28.047502 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:30.137125 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.137063 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 07:03:30.300156 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.300118 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5efe5a5b-a74f-4f75-a733-619770f4790c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5efe5a5b-a74f-4f75-a733-619770f4790c-istiod-ca-cert\") pod \"5efe5a5b-a74f-4f75-a733-619770f4790c\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " Nov 01 07:03:30.300349 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.300183 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-envoy\") pod \"5efe5a5b-a74f-4f75-a733-619770f4790c\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " Nov 01 07:03:30.300349 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.300226 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-workload-certs\") pod \"5efe5a5b-a74f-4f75-a733-619770f4790c\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " Nov 01 07:03:30.300349 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.300306 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-podinfo\") pod \"5efe5a5b-a74f-4f75-a733-619770f4790c\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " Nov 01 07:03:30.300472 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.300397 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-token\" (UniqueName: \"kubernetes.io/projected/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-token\") pod \"5efe5a5b-a74f-4f75-a733-619770f4790c\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " Nov 01 07:03:30.300518 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.300468 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-data\") pod \"5efe5a5b-a74f-4f75-a733-619770f4790c\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " Nov 01 07:03:30.300581 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.300554 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5efe5a5b-a74f-4f75-a733-619770f4790c-kube-api-access-j54tq\" (UniqueName: \"kubernetes.io/projected/5efe5a5b-a74f-4f75-a733-619770f4790c-kube-api-access-j54tq\") pod \"5efe5a5b-a74f-4f75-a733-619770f4790c\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " Nov 01 07:03:30.300638 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.300600 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-workload-socket\") pod \"5efe5a5b-a74f-4f75-a733-619770f4790c\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " Nov 01 07:03:30.300638 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.300604 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-workload-certs" pod "5efe5a5b-a74f-4f75-a733-619770f4790c" (UID: "5efe5a5b-a74f-4f75-a733-619770f4790c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:03:30.300754 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.300632 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5efe5a5b-a74f-4f75-a733-619770f4790c-istiod-ca-cert" pod "5efe5a5b-a74f-4f75-a733-619770f4790c" (UID: "5efe5a5b-a74f-4f75-a733-619770f4790c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:03:30.300754 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.300641 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-credential-socket\") pod \"5efe5a5b-a74f-4f75-a733-619770f4790c\" (UID: \"5efe5a5b-a74f-4f75-a733-619770f4790c\") " Nov 01 07:03:30.300911 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.300802 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-data" pod "5efe5a5b-a74f-4f75-a733-619770f4790c" (UID: "5efe5a5b-a74f-4f75-a733-619770f4790c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:03:30.300911 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.300820 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-workload-socket" pod "5efe5a5b-a74f-4f75-a733-619770f4790c" (UID: "5efe5a5b-a74f-4f75-a733-619770f4790c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:03:30.301045 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.301018 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-credential-socket" pod "5efe5a5b-a74f-4f75-a733-619770f4790c" (UID: "5efe5a5b-a74f-4f75-a733-619770f4790c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:03:30.301397 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.301257 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:30.301397 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.301281 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:30.301397 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.301295 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5efe5a5b-a74f-4f75-a733-619770f4790c-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:30.301397 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.301309 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:30.301397 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.301322 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:30.303373 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.303349 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-envoy" pod "5efe5a5b-a74f-4f75-a733-619770f4790c" (UID: "5efe5a5b-a74f-4f75-a733-619770f4790c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:03:30.303373 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.303360 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-podinfo" pod "5efe5a5b-a74f-4f75-a733-619770f4790c" (UID: "5efe5a5b-a74f-4f75-a733-619770f4790c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:03:30.304029 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.303975 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-token" pod "5efe5a5b-a74f-4f75-a733-619770f4790c" (UID: "5efe5a5b-a74f-4f75-a733-619770f4790c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:03:30.305926 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.305876 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5efe5a5b-a74f-4f75-a733-619770f4790c-kube-api-access-j54tq" pod "5efe5a5b-a74f-4f75-a733-619770f4790c" (UID: "5efe5a5b-a74f-4f75-a733-619770f4790c"). InnerVolumeSpecName "kube-api-access-j54tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:03:30.402724 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.402698 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:30.402724 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.402726 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:30.402933 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.402735 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5efe5a5b-a74f-4f75-a733-619770f4790c-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:30.402933 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.402746 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j54tq\" (UniqueName: \"kubernetes.io/projected/5efe5a5b-a74f-4f75-a733-619770f4790c-kube-api-access-j54tq\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:30.962199 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.962168 4385 generic.go:356] "Generic (PLEG): container finished" podID="5efe5a5b-a74f-4f75-a733-619770f4790c" containerID="48860f04357f0c4a9a02c116f62113329a6e3163606529d247979b9467b61841" exitCode=0 Nov 01 07:03:30.962404 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.962215 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" Nov 01 07:03:30.962404 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.962252 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" event={"ID":"5efe5a5b-a74f-4f75-a733-619770f4790c","Type":"ContainerDied","Data":"48860f04357f0c4a9a02c116f62113329a6e3163606529d247979b9467b61841"} Nov 01 07:03:30.962404 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.962305 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9" event={"ID":"5efe5a5b-a74f-4f75-a733-619770f4790c","Type":"ContainerDied","Data":"9dba9fcbf10c1a3c4f08e1f0c8c5acb1f1d85a50b81ef3252438bbe1de3f22a3"} Nov 01 07:03:30.962404 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.962330 4385 scope.go:122] "RemoveContainer" containerID="48860f04357f0c4a9a02c116f62113329a6e3163606529d247979b9467b61841" Nov 01 07:03:30.972942 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.972921 4385 scope.go:122] "RemoveContainer" containerID="48860f04357f0c4a9a02c116f62113329a6e3163606529d247979b9467b61841" Nov 01 07:03:30.973259 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:03:30.973236 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48860f04357f0c4a9a02c116f62113329a6e3163606529d247979b9467b61841\": container with ID starting with 48860f04357f0c4a9a02c116f62113329a6e3163606529d247979b9467b61841 not found: ID does not exist" containerID="48860f04357f0c4a9a02c116f62113329a6e3163606529d247979b9467b61841" Nov 01 07:03:30.973320 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:30.973271 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48860f04357f0c4a9a02c116f62113329a6e3163606529d247979b9467b61841"} err="failed to get container status \"48860f04357f0c4a9a02c116f62113329a6e3163606529d247979b9467b61841\": rpc error: code = NotFound desc = could not find container \"48860f04357f0c4a9a02c116f62113329a6e3163606529d247979b9467b61841\": container with ID starting with 48860f04357f0c4a9a02c116f62113329a6e3163606529d247979b9467b61841 not found: ID does not exist" Nov 01 07:03:31.177160 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:31.177126 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9"] Nov 01 07:03:31.180413 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:31.180390 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3db672-openshift-default-c799448c-ftpr9"] Nov 01 07:03:31.227187 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:31.227100 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7"] Nov 01 07:03:31.466807 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:31.466775 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:31.613424 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:31.613389 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:31.760090 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:31.760059 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:32.243589 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:32.243525 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:32.389646 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:32.389606 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5efe5a5b-a74f-4f75-a733-619770f4790c" path="/var/lib/kubelet/pods/5efe5a5b-a74f-4f75-a733-619770f4790c/volumes" Nov 01 07:03:34.161110 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:34.161077 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:36.228503 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:36.228421 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7" podUID="0a880bf2-33b3-4d7a-adfa-ed245baa4f9f" containerName="payload-pre-processing" containerID="cri-o://be2cca507693adba475bb0aa0dae6f322898904d5db1c256023cf0fe6b33e33e" gracePeriod=25 Nov 01 07:03:36.513741 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:36.513711 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7" Nov 01 07:03:36.618369 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:36.618330 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a880bf2-33b3-4d7a-adfa-ed245baa4f9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a880bf2-33b3-4d7a-adfa-ed245baa4f9f-plugins-config-volume\") pod \"0a880bf2-33b3-4d7a-adfa-ed245baa4f9f\" (UID: \"0a880bf2-33b3-4d7a-adfa-ed245baa4f9f\") " Nov 01 07:03:36.618709 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:36.618690 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a880bf2-33b3-4d7a-adfa-ed245baa4f9f-kube-api-access-5nrf7\" (UniqueName: \"kubernetes.io/projected/0a880bf2-33b3-4d7a-adfa-ed245baa4f9f-kube-api-access-5nrf7\") pod \"0a880bf2-33b3-4d7a-adfa-ed245baa4f9f\" (UID: \"0a880bf2-33b3-4d7a-adfa-ed245baa4f9f\") " Nov 01 07:03:36.621876 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:36.619601 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a880bf2-33b3-4d7a-adfa-ed245baa4f9f-plugins-config-volume" pod "0a880bf2-33b3-4d7a-adfa-ed245baa4f9f" (UID: "0a880bf2-33b3-4d7a-adfa-ed245baa4f9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:03:36.622554 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:36.622266 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a880bf2-33b3-4d7a-adfa-ed245baa4f9f-kube-api-access-5nrf7" pod "0a880bf2-33b3-4d7a-adfa-ed245baa4f9f" (UID: "0a880bf2-33b3-4d7a-adfa-ed245baa4f9f"). InnerVolumeSpecName "kube-api-access-5nrf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:03:36.720895 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:36.720861 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a880bf2-33b3-4d7a-adfa-ed245baa4f9f-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:36.720895 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:36.720895 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nrf7\" (UniqueName: \"kubernetes.io/projected/0a880bf2-33b3-4d7a-adfa-ed245baa4f9f-kube-api-access-5nrf7\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:36.779184 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:03:36.779105 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:03:37.004938 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:37.004906 4385 generic.go:356] "Generic (PLEG): container finished" podID="0a880bf2-33b3-4d7a-adfa-ed245baa4f9f" containerID="be2cca507693adba475bb0aa0dae6f322898904d5db1c256023cf0fe6b33e33e" exitCode=0 Nov 01 07:03:37.005117 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:37.004962 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7" Nov 01 07:03:37.005117 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:37.004993 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7" event={"ID":"0a880bf2-33b3-4d7a-adfa-ed245baa4f9f","Type":"ContainerDied","Data":"be2cca507693adba475bb0aa0dae6f322898904d5db1c256023cf0fe6b33e33e"} Nov 01 07:03:37.005117 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:37.005048 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7" event={"ID":"0a880bf2-33b3-4d7a-adfa-ed245baa4f9f","Type":"ContainerDied","Data":"ee59b17a444d8b1f334453117f1594d97fbc38dd7a63b042b1f0684dc201a625"} Nov 01 07:03:37.005117 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:37.005069 4385 scope.go:122] "RemoveContainer" containerID="be2cca507693adba475bb0aa0dae6f322898904d5db1c256023cf0fe6b33e33e" Nov 01 07:03:37.015740 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:37.015722 4385 scope.go:122] "RemoveContainer" containerID="be2cca507693adba475bb0aa0dae6f322898904d5db1c256023cf0fe6b33e33e" Nov 01 07:03:37.016071 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:03:37.016052 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be2cca507693adba475bb0aa0dae6f322898904d5db1c256023cf0fe6b33e33e\": container with ID starting with be2cca507693adba475bb0aa0dae6f322898904d5db1c256023cf0fe6b33e33e not found: ID does not exist" containerID="be2cca507693adba475bb0aa0dae6f322898904d5db1c256023cf0fe6b33e33e" Nov 01 07:03:37.016138 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:37.016081 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be2cca507693adba475bb0aa0dae6f322898904d5db1c256023cf0fe6b33e33e"} err="failed to get container status \"be2cca507693adba475bb0aa0dae6f322898904d5db1c256023cf0fe6b33e33e\": rpc error: code = NotFound desc = could not find container \"be2cca507693adba475bb0aa0dae6f322898904d5db1c256023cf0fe6b33e33e\": container with ID starting with be2cca507693adba475bb0aa0dae6f322898904d5db1c256023cf0fe6b33e33e not found: ID does not exist" Nov 01 07:03:37.169226 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:37.169190 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7"] Nov 01 07:03:37.218326 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:37.218291 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c52bfa-7f95745579-kvlp7"] Nov 01 07:03:37.258641 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:37.258606 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:37.456483 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:37.456369 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:38.384868 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:38.384824 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a880bf2-33b3-4d7a-adfa-ed245baa4f9f" path="/var/lib/kubelet/pods/0a880bf2-33b3-4d7a-adfa-ed245baa4f9f/volumes" Nov 01 07:03:38.795039 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:38.794948 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:39.259755 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:39.259720 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:39.416369 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:39.416274 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:39.803586 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:39.803553 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:39.960332 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:39.960297 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:40.118985 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:40.118951 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:40.378894 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:40.378779 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:41.190376 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:41.190335 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:41.346985 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:41.346941 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:43.625810 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:43.625776 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6"] Nov 01 07:03:43.626246 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:43.626016 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" podUID="c42d21f9-d165-4278-b571-678356d3906e" containerName="maas-api" containerID="cri-o://30165fef155e1487a939819759bef5c02bf02712da69501fc275bc34b86523a1" gracePeriod=30 Nov 01 07:03:43.749535 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:43.749499 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:43.942187 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:43.942158 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" Nov 01 07:03:44.018641 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.017317 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c42d21f9-d165-4278-b571-678356d3906e-kube-api-access-sgp28\" (UniqueName: \"kubernetes.io/projected/c42d21f9-d165-4278-b571-678356d3906e-kube-api-access-sgp28\") pod \"c42d21f9-d165-4278-b571-678356d3906e\" (UID: \"c42d21f9-d165-4278-b571-678356d3906e\") " Nov 01 07:03:44.018641 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.017404 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c42d21f9-d165-4278-b571-678356d3906e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c42d21f9-d165-4278-b571-678356d3906e-metrics-certs\") pod \"c42d21f9-d165-4278-b571-678356d3906e\" (UID: \"c42d21f9-d165-4278-b571-678356d3906e\") " Nov 01 07:03:44.018641 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.017454 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c42d21f9-d165-4278-b571-678356d3906e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42d21f9-d165-4278-b571-678356d3906e-openshift-service-ca\") pod \"c42d21f9-d165-4278-b571-678356d3906e\" (UID: \"c42d21f9-d165-4278-b571-678356d3906e\") " Nov 01 07:03:44.018641 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.017514 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c42d21f9-d165-4278-b571-678356d3906e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c42d21f9-d165-4278-b571-678356d3906e-maas-api-tls\") pod \"c42d21f9-d165-4278-b571-678356d3906e\" (UID: \"c42d21f9-d165-4278-b571-678356d3906e\") " Nov 01 07:03:44.020203 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.020169 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42d21f9-d165-4278-b571-678356d3906e-openshift-service-ca" pod "c42d21f9-d165-4278-b571-678356d3906e" (UID: "c42d21f9-d165-4278-b571-678356d3906e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:03:44.031213 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.031145 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42d21f9-d165-4278-b571-678356d3906e-maas-api-tls" pod "c42d21f9-d165-4278-b571-678356d3906e" (UID: "c42d21f9-d165-4278-b571-678356d3906e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:03:44.035037 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.034983 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42d21f9-d165-4278-b571-678356d3906e-kube-api-access-sgp28" pod "c42d21f9-d165-4278-b571-678356d3906e" (UID: "c42d21f9-d165-4278-b571-678356d3906e"). InnerVolumeSpecName "kube-api-access-sgp28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:03:44.035620 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.035576 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42d21f9-d165-4278-b571-678356d3906e-metrics-certs" pod "c42d21f9-d165-4278-b571-678356d3906e" (UID: "c42d21f9-d165-4278-b571-678356d3906e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:03:44.063583 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.063549 4385 generic.go:356] "Generic (PLEG): container finished" podID="c42d21f9-d165-4278-b571-678356d3906e" containerID="30165fef155e1487a939819759bef5c02bf02712da69501fc275bc34b86523a1" exitCode=0 Nov 01 07:03:44.063775 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.063620 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" Nov 01 07:03:44.063775 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.063635 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" event={"ID":"c42d21f9-d165-4278-b571-678356d3906e","Type":"ContainerDied","Data":"30165fef155e1487a939819759bef5c02bf02712da69501fc275bc34b86523a1"} Nov 01 07:03:44.063775 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.063707 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6" event={"ID":"c42d21f9-d165-4278-b571-678356d3906e","Type":"ContainerDied","Data":"4854f81c4b208d4bb4ff16e514875db7f4982afd77705341481231a0257fb43c"} Nov 01 07:03:44.063775 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.063736 4385 scope.go:122] "RemoveContainer" containerID="30165fef155e1487a939819759bef5c02bf02712da69501fc275bc34b86523a1" Nov 01 07:03:44.076084 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.075816 4385 scope.go:122] "RemoveContainer" containerID="30165fef155e1487a939819759bef5c02bf02712da69501fc275bc34b86523a1" Nov 01 07:03:44.076342 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:03:44.076316 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30165fef155e1487a939819759bef5c02bf02712da69501fc275bc34b86523a1\": container with ID starting with 30165fef155e1487a939819759bef5c02bf02712da69501fc275bc34b86523a1 not found: ID does not exist" containerID="30165fef155e1487a939819759bef5c02bf02712da69501fc275bc34b86523a1" Nov 01 07:03:44.076421 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.076351 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30165fef155e1487a939819759bef5c02bf02712da69501fc275bc34b86523a1"} err="failed to get container status \"30165fef155e1487a939819759bef5c02bf02712da69501fc275bc34b86523a1\": rpc error: code = NotFound desc = could not find container \"30165fef155e1487a939819759bef5c02bf02712da69501fc275bc34b86523a1\": container with ID starting with 30165fef155e1487a939819759bef5c02bf02712da69501fc275bc34b86523a1 not found: ID does not exist" Nov 01 07:03:44.119433 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.119392 4385 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c42d21f9-d165-4278-b571-678356d3906e-metrics-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:44.119433 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.119430 4385 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42d21f9-d165-4278-b571-678356d3906e-openshift-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:44.119695 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.119443 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c42d21f9-d165-4278-b571-678356d3906e-maas-api-tls\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:44.119695 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.119456 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgp28\" (UniqueName: \"kubernetes.io/projected/c42d21f9-d165-4278-b571-678356d3906e-kube-api-access-sgp28\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:44.228710 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.228635 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6"] Nov 01 07:03:44.233914 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.233887 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-9dbr6"] Nov 01 07:03:44.386662 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:44.386630 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c42d21f9-d165-4278-b571-678356d3906e" path="/var/lib/kubelet/pods/c42d21f9-d165-4278-b571-678356d3906e/volumes" Nov 01 07:03:45.224294 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:45.224262 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:45.382288 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:45.382255 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:45.635260 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:45.635225 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:49.309530 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:49.309480 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:50.516557 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:50.516525 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:50.882950 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:50.882920 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:52.116366 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:52.116274 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:52.832352 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:52.832322 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:52.979487 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:52.979450 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:55.099947 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:55.099894 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:55.294907 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:55.294869 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:55.556985 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:55.556883 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:56.324452 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:56.324415 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:03:56.768429 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:03:56.768344 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:00.296754 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:00.296697 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:02.137945 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:02.137909 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x"] Nov 01 07:04:02.160632 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:02.160600 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:02.890277 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:02.890236 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:03.851002 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:03.850961 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk"] Nov 01 07:04:04.216174 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:04.216085 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:04.360172 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:04.360138 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:05.039884 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:05.039814 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:06.779544 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:04:06.779507 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:04:08.328588 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:08.328547 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:08.851927 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:08.851876 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk" podUID="4be0c016-1f2d-4be4-a99a-ced55c192443" containerName="payload-processing" containerID="cri-o://a91912e5ae68959b8b969d85947253e66f93e46e5d50216156bab7536fd0def3" gracePeriod=25 Nov 01 07:04:09.825901 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:09.825865 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:12.145390 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:12.145342 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk" podUID="4be0c016-1f2d-4be4-a99a-ced55c192443" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:04:13.948981 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:13.948944 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:15.715092 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:15.715043 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:17.159473 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.159429 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" podUID="4f5b6430-dcf4-429c-bddc-f50bfba85c8f" containerName="main" containerID="cri-o://d1eb3cf9f68d099b7fb9f63c39a3a885d18fd55ba906088ed28ffb0128295be5" gracePeriod=45 Nov 01 07:04:17.372394 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.372357 4385 generic.go:356] "Generic (PLEG): container finished" podID="4f5b6430-dcf4-429c-bddc-f50bfba85c8f" containerID="d1eb3cf9f68d099b7fb9f63c39a3a885d18fd55ba906088ed28ffb0128295be5" exitCode=0 Nov 01 07:04:17.372585 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.372436 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" event={"ID":"4f5b6430-dcf4-429c-bddc-f50bfba85c8f","Type":"ContainerDied","Data":"d1eb3cf9f68d099b7fb9f63c39a3a885d18fd55ba906088ed28ffb0128295be5"} Nov 01 07:04:17.432484 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.432457 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 07:04:17.504280 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.504251 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-dshm\") pod \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " Nov 01 07:04:17.504463 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.504290 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-tls-certs\") pod \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " Nov 01 07:04:17.504463 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.504347 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-model-cache\") pod \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " Nov 01 07:04:17.504463 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.504375 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-tmp-dir\") pod \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " Nov 01 07:04:17.504463 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.504419 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-kube-api-access-wgf7v\" (UniqueName: \"kubernetes.io/projected/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-kube-api-access-wgf7v\") pod \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " Nov 01 07:04:17.505004 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.504494 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-home\" (UniqueName: \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-home\") pod \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\" (UID: \"4f5b6430-dcf4-429c-bddc-f50bfba85c8f\") " Nov 01 07:04:17.505004 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.504661 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-model-cache" pod "4f5b6430-dcf4-429c-bddc-f50bfba85c8f" (UID: "4f5b6430-dcf4-429c-bddc-f50bfba85c8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:04:17.505004 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.504693 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-tmp-dir" pod "4f5b6430-dcf4-429c-bddc-f50bfba85c8f" (UID: "4f5b6430-dcf4-429c-bddc-f50bfba85c8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:04:17.505004 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.504968 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-home" pod "4f5b6430-dcf4-429c-bddc-f50bfba85c8f" (UID: "4f5b6430-dcf4-429c-bddc-f50bfba85c8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:04:17.505396 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.505330 4385 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-model-cache\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:17.505396 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.505355 4385 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-tmp-dir\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:17.505396 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.505369 4385 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-home\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:17.507052 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.507028 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-kube-api-access-wgf7v" pod "4f5b6430-dcf4-429c-bddc-f50bfba85c8f" (UID: "4f5b6430-dcf4-429c-bddc-f50bfba85c8f"). InnerVolumeSpecName "kube-api-access-wgf7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:04:17.507240 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.507219 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-dshm" pod "4f5b6430-dcf4-429c-bddc-f50bfba85c8f" (UID: "4f5b6430-dcf4-429c-bddc-f50bfba85c8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:04:17.507323 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.507238 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-tls-certs" pod "4f5b6430-dcf4-429c-bddc-f50bfba85c8f" (UID: "4f5b6430-dcf4-429c-bddc-f50bfba85c8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:04:17.608109 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.608075 4385 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-dshm\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:17.608471 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.608381 4385 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-tls-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:17.608657 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:17.608640 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgf7v\" (UniqueName: \"kubernetes.io/projected/4f5b6430-dcf4-429c-bddc-f50bfba85c8f-kube-api-access-wgf7v\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:18.380737 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:18.380710 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" Nov 01 07:04:18.384760 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:18.384729 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" event={"ID":"4f5b6430-dcf4-429c-bddc-f50bfba85c8f","Type":"ContainerDied","Data":"bc3b22e2761ab67f4ef019f659af02780182f5b0416734b61ae167853b0f1549"} Nov 01 07:04:18.384760 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:18.384771 4385 scope.go:122] "RemoveContainer" containerID="d1eb3cf9f68d099b7fb9f63c39a3a885d18fd55ba906088ed28ffb0128295be5" Nov 01 07:04:18.427818 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:18.427782 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x" podUID="4f5b6430-dcf4-429c-bddc-f50bfba85c8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.147:8000/ready\": context deadline exceeded" Nov 01 07:04:18.558734 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:18.558702 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x"] Nov 01 07:04:18.573726 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:18.573697 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-302ba0/sub-test-model-302ba0-kserve-74c4f66bdf-cjl9x"] Nov 01 07:04:20.386541 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:20.386505 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f5b6430-dcf4-429c-bddc-f50bfba85c8f" path="/var/lib/kubelet/pods/4f5b6430-dcf4-429c-bddc-f50bfba85c8f/volumes" Nov 01 07:04:20.418031 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:20.417998 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:22.145489 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:22.145444 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk" podUID="4be0c016-1f2d-4be4-a99a-ced55c192443" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:04:22.626182 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:22.626140 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:22.775530 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:22.775490 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:24.118689 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:24.118657 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:24.622817 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:24.622786 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:25.352648 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:25.352618 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:25.495296 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:25.495259 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:28.522417 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:28.522374 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:29.140679 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:29.140655 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk" Nov 01 07:04:29.243985 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:29.243957 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be0c016-1f2d-4be4-a99a-ced55c192443-kube-api-access-jv25s\" (UniqueName: \"kubernetes.io/projected/4be0c016-1f2d-4be4-a99a-ced55c192443-kube-api-access-jv25s\") pod \"4be0c016-1f2d-4be4-a99a-ced55c192443\" (UID: \"4be0c016-1f2d-4be4-a99a-ced55c192443\") " Nov 01 07:04:29.244179 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:29.244031 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4be0c016-1f2d-4be4-a99a-ced55c192443-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be0c016-1f2d-4be4-a99a-ced55c192443-plugins-config-volume\") pod \"4be0c016-1f2d-4be4-a99a-ced55c192443\" (UID: \"4be0c016-1f2d-4be4-a99a-ced55c192443\") " Nov 01 07:04:29.244429 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:29.244405 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be0c016-1f2d-4be4-a99a-ced55c192443-plugins-config-volume" pod "4be0c016-1f2d-4be4-a99a-ced55c192443" (UID: "4be0c016-1f2d-4be4-a99a-ced55c192443"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:04:29.246345 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:29.246317 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be0c016-1f2d-4be4-a99a-ced55c192443-kube-api-access-jv25s" pod "4be0c016-1f2d-4be4-a99a-ced55c192443" (UID: "4be0c016-1f2d-4be4-a99a-ced55c192443"). InnerVolumeSpecName "kube-api-access-jv25s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:04:29.345707 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:29.345666 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv25s\" (UniqueName: \"kubernetes.io/projected/4be0c016-1f2d-4be4-a99a-ced55c192443-kube-api-access-jv25s\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:29.345707 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:29.345710 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be0c016-1f2d-4be4-a99a-ced55c192443-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:29.457323 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:29.457242 4385 generic.go:356] "Generic (PLEG): container finished" podID="4be0c016-1f2d-4be4-a99a-ced55c192443" containerID="a91912e5ae68959b8b969d85947253e66f93e46e5d50216156bab7536fd0def3" exitCode=0 Nov 01 07:04:29.457323 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:29.457288 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk" Nov 01 07:04:29.457513 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:29.457331 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk" event={"ID":"4be0c016-1f2d-4be4-a99a-ced55c192443","Type":"ContainerDied","Data":"a91912e5ae68959b8b969d85947253e66f93e46e5d50216156bab7536fd0def3"} Nov 01 07:04:29.457513 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:29.457385 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk" event={"ID":"4be0c016-1f2d-4be4-a99a-ced55c192443","Type":"ContainerDied","Data":"e75fcba4a49147328191653599ac25b853cb0501a4aaf88f123c403756772a96"} Nov 01 07:04:29.457513 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:29.457408 4385 scope.go:122] "RemoveContainer" containerID="a91912e5ae68959b8b969d85947253e66f93e46e5d50216156bab7536fd0def3" Nov 01 07:04:29.468139 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:29.468115 4385 scope.go:122] "RemoveContainer" containerID="a91912e5ae68959b8b969d85947253e66f93e46e5d50216156bab7536fd0def3" Nov 01 07:04:29.468467 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:04:29.468448 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a91912e5ae68959b8b969d85947253e66f93e46e5d50216156bab7536fd0def3\": container with ID starting with a91912e5ae68959b8b969d85947253e66f93e46e5d50216156bab7536fd0def3 not found: ID does not exist" containerID="a91912e5ae68959b8b969d85947253e66f93e46e5d50216156bab7536fd0def3" Nov 01 07:04:29.468545 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:29.468475 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a91912e5ae68959b8b969d85947253e66f93e46e5d50216156bab7536fd0def3"} err="failed to get container status \"a91912e5ae68959b8b969d85947253e66f93e46e5d50216156bab7536fd0def3\": rpc error: code = NotFound desc = could not find container \"a91912e5ae68959b8b969d85947253e66f93e46e5d50216156bab7536fd0def3\": container with ID starting with a91912e5ae68959b8b969d85947253e66f93e46e5d50216156bab7536fd0def3 not found: ID does not exist" Nov 01 07:04:29.621362 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:29.621331 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk"] Nov 01 07:04:29.624602 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:29.624579 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c79abe-5dc676ccd5-kf2pk"] Nov 01 07:04:30.160177 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:30.160140 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:30.386061 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:30.386022 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4be0c016-1f2d-4be4-a99a-ced55c192443" path="/var/lib/kubelet/pods/4be0c016-1f2d-4be4-a99a-ced55c192443/volumes" Nov 01 07:04:31.313247 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:31.313031 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:32.195818 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:32.195781 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:32.391766 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:32.391728 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:35.114860 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:35.114788 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:36.781083 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:04:36.781050 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:04:37.030599 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:37.030561 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:39.172680 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:39.172638 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:40.916444 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:40.916408 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:42.574967 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:42.574926 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:42.737096 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:42.737057 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:44.093729 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:44.093687 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:44.570054 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:44.570008 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:45.072500 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:45.072467 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6"] Nov 01 07:04:45.105198 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:45.105159 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:46.017474 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:46.017440 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:47.464934 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:47.464902 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:50.690222 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:50.690168 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:52.072315 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:52.072269 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:52.458255 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:52.458200 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:04:55.531351 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:04:55.531293 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:05:00.093826 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.093776 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" podUID="67193eb4-e54e-4b57-b9f1-6cf576af70f7" containerName="main" containerID="cri-o://375ea6b05e18f40ee83eb05b75b219f0e5da780bab11de27f66f3e5492ae734c" gracePeriod=45 Nov 01 07:05:00.231382 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.231344 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:05:00.386268 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.386238 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 07:05:00.410727 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.410688 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:05:00.521076 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.521035 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-home\" (UniqueName: \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-home\") pod \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " Nov 01 07:05:00.521283 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.521105 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67193eb4-e54e-4b57-b9f1-6cf576af70f7-kube-api-access-bnxl6\" (UniqueName: \"kubernetes.io/projected/67193eb4-e54e-4b57-b9f1-6cf576af70f7-kube-api-access-bnxl6\") pod \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " Nov 01 07:05:00.521283 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.521168 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-model-cache\") pod \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " Nov 01 07:05:00.521419 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.521340 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-home" pod "67193eb4-e54e-4b57-b9f1-6cf576af70f7" (UID: "67193eb4-e54e-4b57-b9f1-6cf576af70f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:05:00.521419 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.521370 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-tmp-dir\") pod \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " Nov 01 07:05:00.521535 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.521442 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67193eb4-e54e-4b57-b9f1-6cf576af70f7-tls-certs\" (UniqueName: \"kubernetes.io/secret/67193eb4-e54e-4b57-b9f1-6cf576af70f7-tls-certs\") pod \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " Nov 01 07:05:00.521535 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.521456 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-model-cache" pod "67193eb4-e54e-4b57-b9f1-6cf576af70f7" (UID: "67193eb4-e54e-4b57-b9f1-6cf576af70f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:05:00.521651 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.521538 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-dshm\") pod \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\" (UID: \"67193eb4-e54e-4b57-b9f1-6cf576af70f7\") " Nov 01 07:05:00.521707 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.521540 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-tmp-dir" pod "67193eb4-e54e-4b57-b9f1-6cf576af70f7" (UID: "67193eb4-e54e-4b57-b9f1-6cf576af70f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:05:00.522094 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.522069 4385 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-home\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:05:00.522246 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.522098 4385 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-model-cache\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:05:00.522246 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.522115 4385 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-tmp-dir\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:05:00.523955 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.523927 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-dshm" pod "67193eb4-e54e-4b57-b9f1-6cf576af70f7" (UID: "67193eb4-e54e-4b57-b9f1-6cf576af70f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:05:00.524391 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.524365 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67193eb4-e54e-4b57-b9f1-6cf576af70f7-kube-api-access-bnxl6" pod "67193eb4-e54e-4b57-b9f1-6cf576af70f7" (UID: "67193eb4-e54e-4b57-b9f1-6cf576af70f7"). InnerVolumeSpecName "kube-api-access-bnxl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:05:00.524391 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.524389 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67193eb4-e54e-4b57-b9f1-6cf576af70f7-tls-certs" pod "67193eb4-e54e-4b57-b9f1-6cf576af70f7" (UID: "67193eb4-e54e-4b57-b9f1-6cf576af70f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:05:00.623748 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.623671 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnxl6\" (UniqueName: \"kubernetes.io/projected/67193eb4-e54e-4b57-b9f1-6cf576af70f7-kube-api-access-bnxl6\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:05:00.623748 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.623699 4385 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67193eb4-e54e-4b57-b9f1-6cf576af70f7-tls-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:05:00.623748 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.623709 4385 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67193eb4-e54e-4b57-b9f1-6cf576af70f7-dshm\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:05:00.681214 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.681175 4385 generic.go:356] "Generic (PLEG): container finished" podID="67193eb4-e54e-4b57-b9f1-6cf576af70f7" containerID="375ea6b05e18f40ee83eb05b75b219f0e5da780bab11de27f66f3e5492ae734c" exitCode=0 Nov 01 07:05:00.681419 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.681236 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" Nov 01 07:05:00.681419 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.681256 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" event={"ID":"67193eb4-e54e-4b57-b9f1-6cf576af70f7","Type":"ContainerDied","Data":"375ea6b05e18f40ee83eb05b75b219f0e5da780bab11de27f66f3e5492ae734c"} Nov 01 07:05:00.681419 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.681312 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6" event={"ID":"67193eb4-e54e-4b57-b9f1-6cf576af70f7","Type":"ContainerDied","Data":"7535425b5544cfc5f35c2982da2471f7b493e44e8e4d5c3f63b5a4ac9e8fd7a2"} Nov 01 07:05:00.681419 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.681334 4385 scope.go:122] "RemoveContainer" containerID="375ea6b05e18f40ee83eb05b75b219f0e5da780bab11de27f66f3e5492ae734c" Nov 01 07:05:00.694496 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.694476 4385 scope.go:122] "RemoveContainer" containerID="375ea6b05e18f40ee83eb05b75b219f0e5da780bab11de27f66f3e5492ae734c" Nov 01 07:05:00.694870 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:05:00.694811 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"375ea6b05e18f40ee83eb05b75b219f0e5da780bab11de27f66f3e5492ae734c\": container with ID starting with 375ea6b05e18f40ee83eb05b75b219f0e5da780bab11de27f66f3e5492ae734c not found: ID does not exist" containerID="375ea6b05e18f40ee83eb05b75b219f0e5da780bab11de27f66f3e5492ae734c" Nov 01 07:05:00.694950 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.694877 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"375ea6b05e18f40ee83eb05b75b219f0e5da780bab11de27f66f3e5492ae734c"} err="failed to get container status \"375ea6b05e18f40ee83eb05b75b219f0e5da780bab11de27f66f3e5492ae734c\": rpc error: code = NotFound desc = could not find container \"375ea6b05e18f40ee83eb05b75b219f0e5da780bab11de27f66f3e5492ae734c\": container with ID starting with 375ea6b05e18f40ee83eb05b75b219f0e5da780bab11de27f66f3e5492ae734c not found: ID does not exist" Nov 01 07:05:00.845168 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.845138 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6"] Nov 01 07:05:00.848570 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:00.848547 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-6ba6e5/sub-test-model-6ba6e5-kserve-6b4f6b9d79-88vl6"] Nov 01 07:05:02.387262 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:02.387230 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67193eb4-e54e-4b57-b9f1-6cf576af70f7" path="/var/lib/kubelet/pods/67193eb4-e54e-4b57-b9f1-6cf576af70f7/volumes" Nov 01 07:05:04.929666 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:04.929632 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:05:06.784127 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:05:06.784085 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:05:09.762374 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:09.762332 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:05:09.953639 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:09.953602 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:05:13.074147 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:13.074104 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:05:14.441357 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:14.441310 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:05:19.117767 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:19.117731 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:05:23.819397 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:23.819362 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:05:28.542413 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:28.542373 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:05:30.142195 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142157 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb"] Nov 01 07:05:30.142672 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142657 4385 cpu_manager.go:414] "containerMap: removing container" podUID="5efe5a5b-a74f-4f75-a733-619770f4790c" containerName="istio-proxy" Nov 01 07:05:30.142719 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142673 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5efe5a5b-a74f-4f75-a733-619770f4790c" containerName="istio-proxy" Nov 01 07:05:30.142719 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142687 4385 cpu_manager.go:414] "containerMap: removing container" podUID="3871d20b-6d9f-4c6e-87b1-8e66cc4ef350" containerName="cleanup" Nov 01 07:05:30.142719 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142692 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3871d20b-6d9f-4c6e-87b1-8e66cc4ef350" containerName="cleanup" Nov 01 07:05:30.142719 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142701 4385 cpu_manager.go:414] "containerMap: removing container" podUID="c42d21f9-d165-4278-b571-678356d3906e" containerName="maas-api" Nov 01 07:05:30.142719 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142706 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42d21f9-d165-4278-b571-678356d3906e" containerName="maas-api" Nov 01 07:05:30.142719 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142713 4385 cpu_manager.go:414] "containerMap: removing container" podUID="3871d20b-6d9f-4c6e-87b1-8e66cc4ef350" containerName="cleanup" Nov 01 07:05:30.142719 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142718 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3871d20b-6d9f-4c6e-87b1-8e66cc4ef350" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142732 4385 cpu_manager.go:414] "containerMap: removing container" podUID="247e6488-ee32-4931-9670-ea759c4cc896" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142737 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247e6488-ee32-4931-9670-ea759c4cc896" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142747 4385 cpu_manager.go:414] "containerMap: removing container" podUID="d27e5671-8155-4605-9333-001d14195f62" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142754 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d27e5671-8155-4605-9333-001d14195f62" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142766 4385 cpu_manager.go:414] "containerMap: removing container" podUID="0a880bf2-33b3-4d7a-adfa-ed245baa4f9f" containerName="payload-pre-processing" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142770 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a880bf2-33b3-4d7a-adfa-ed245baa4f9f" containerName="payload-pre-processing" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142778 4385 cpu_manager.go:414] "containerMap: removing container" podUID="becaf42b-35ef-46db-a4f8-0cc8780722f0" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142782 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="becaf42b-35ef-46db-a4f8-0cc8780722f0" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142787 4385 cpu_manager.go:414] "containerMap: removing container" podUID="9debfa95-4fbc-4962-8177-6a53821135ca" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142792 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9debfa95-4fbc-4962-8177-6a53821135ca" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142800 4385 cpu_manager.go:414] "containerMap: removing container" podUID="f3b1e9b1-6c86-4818-bab1-715cad35fb8c" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142805 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b1e9b1-6c86-4818-bab1-715cad35fb8c" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142811 4385 cpu_manager.go:414] "containerMap: removing container" podUID="4f5b6430-dcf4-429c-bddc-f50bfba85c8f" containerName="main" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142815 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f5b6430-dcf4-429c-bddc-f50bfba85c8f" containerName="main" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142822 4385 cpu_manager.go:414] "containerMap: removing container" podUID="60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142826 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142851 4385 cpu_manager.go:414] "containerMap: removing container" podUID="247e6488-ee32-4931-9670-ea759c4cc896" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142857 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247e6488-ee32-4931-9670-ea759c4cc896" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142865 4385 cpu_manager.go:414] "containerMap: removing container" podUID="9debfa95-4fbc-4962-8177-6a53821135ca" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142869 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9debfa95-4fbc-4962-8177-6a53821135ca" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142876 4385 cpu_manager.go:414] "containerMap: removing container" podUID="d27e5671-8155-4605-9333-001d14195f62" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142880 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d27e5671-8155-4605-9333-001d14195f62" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142887 4385 cpu_manager.go:414] "containerMap: removing container" podUID="d27e5671-8155-4605-9333-001d14195f62" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142891 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d27e5671-8155-4605-9333-001d14195f62" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142900 4385 cpu_manager.go:414] "containerMap: removing container" podUID="25f4c33e-6491-4057-8979-f8ff03d56c6f" containerName="istio-proxy" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142904 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25f4c33e-6491-4057-8979-f8ff03d56c6f" containerName="istio-proxy" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142911 4385 cpu_manager.go:414] "containerMap: removing container" podUID="f3b1e9b1-6c86-4818-bab1-715cad35fb8c" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142915 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b1e9b1-6c86-4818-bab1-715cad35fb8c" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142925 4385 cpu_manager.go:414] "containerMap: removing container" podUID="22d2e118-2604-4c39-9f1a-cf6a509397fa" containerName="payload-pre-processing" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142929 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22d2e118-2604-4c39-9f1a-cf6a509397fa" containerName="payload-pre-processing" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142936 4385 cpu_manager.go:414] "containerMap: removing container" podUID="250ce347-5c48-4490-b942-0f4916c707a5" containerName="payload-processing" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142940 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="250ce347-5c48-4490-b942-0f4916c707a5" containerName="payload-processing" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142948 4385 cpu_manager.go:414] "containerMap: removing container" podUID="4be0c016-1f2d-4be4-a99a-ced55c192443" containerName="payload-processing" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142953 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be0c016-1f2d-4be4-a99a-ced55c192443" containerName="payload-processing" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142960 4385 cpu_manager.go:414] "containerMap: removing container" podUID="60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142964 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142974 4385 cpu_manager.go:414] "containerMap: removing container" podUID="67193eb4-e54e-4b57-b9f1-6cf576af70f7" containerName="main" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142977 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67193eb4-e54e-4b57-b9f1-6cf576af70f7" containerName="main" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142984 4385 cpu_manager.go:414] "containerMap: removing container" podUID="7d064de5-6ab4-45ab-a83a-0fe33a874788" containerName="payload-processing" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142988 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d064de5-6ab4-45ab-a83a-0fe33a874788" containerName="payload-processing" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.142995 4385 cpu_manager.go:414] "containerMap: removing container" podUID="becaf42b-35ef-46db-a4f8-0cc8780722f0" containerName="cleanup" Nov 01 07:05:30.142986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143000 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="becaf42b-35ef-46db-a4f8-0cc8780722f0" containerName="cleanup" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143076 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b1e9b1-6c86-4818-bab1-715cad35fb8c" containerName="cleanup" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143084 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f5b6430-dcf4-429c-bddc-f50bfba85c8f" containerName="main" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143090 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac" containerName="cleanup" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143097 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="247e6488-ee32-4931-9670-ea759c4cc896" containerName="cleanup" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143102 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="9debfa95-4fbc-4962-8177-6a53821135ca" containerName="cleanup" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143108 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="d27e5671-8155-4605-9333-001d14195f62" containerName="cleanup" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143113 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="d27e5671-8155-4605-9333-001d14195f62" containerName="cleanup" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143119 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b1e9b1-6c86-4818-bab1-715cad35fb8c" containerName="cleanup" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143125 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="9debfa95-4fbc-4962-8177-6a53821135ca" containerName="cleanup" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143130 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="25f4c33e-6491-4057-8979-f8ff03d56c6f" containerName="istio-proxy" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143136 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b1e9b1-6c86-4818-bab1-715cad35fb8c" containerName="cleanup" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143141 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="22d2e118-2604-4c39-9f1a-cf6a509397fa" containerName="payload-pre-processing" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143146 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac" containerName="cleanup" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143150 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="250ce347-5c48-4490-b942-0f4916c707a5" containerName="payload-processing" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143156 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be0c016-1f2d-4be4-a99a-ced55c192443" containerName="payload-processing" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143162 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="3871d20b-6d9f-4c6e-87b1-8e66cc4ef350" containerName="cleanup" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143167 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="becaf42b-35ef-46db-a4f8-0cc8780722f0" containerName="cleanup" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143173 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="67193eb4-e54e-4b57-b9f1-6cf576af70f7" containerName="main" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143183 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="247e6488-ee32-4931-9670-ea759c4cc896" containerName="cleanup" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143190 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d064de5-6ab4-45ab-a83a-0fe33a874788" containerName="payload-processing" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143199 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="becaf42b-35ef-46db-a4f8-0cc8780722f0" containerName="cleanup" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143206 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="5efe5a5b-a74f-4f75-a733-619770f4790c" containerName="istio-proxy" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143211 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="c42d21f9-d165-4278-b571-678356d3906e" containerName="maas-api" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143217 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="3871d20b-6d9f-4c6e-87b1-8e66cc4ef350" containerName="cleanup" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143224 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a880bf2-33b3-4d7a-adfa-ed245baa4f9f" containerName="payload-pre-processing" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143229 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="becaf42b-35ef-46db-a4f8-0cc8780722f0" containerName="cleanup" Nov 01 07:05:30.144215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.143235 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="9debfa95-4fbc-4962-8177-6a53821135ca" containerName="cleanup" Nov 01 07:05:30.146543 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.146524 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.198608 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.198570 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c831a4c0-openshift-default-dockercfg-mtmts\"" Nov 01 07:05:30.251174 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.251137 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.251174 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.251183 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.251406 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.251283 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.251445 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.251411 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4vnh\" (UniqueName: \"kubernetes.io/projected/25eb2ba5-e303-4a08-8720-1b60361c2d5e-kube-api-access-t4vnh\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.251487 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.251445 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.251487 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.251465 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.251597 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.251504 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.251597 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.251542 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.251689 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.251612 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.352311 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.352272 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb"] Nov 01 07:05:30.352545 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.352523 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vnh\" (UniqueName: \"kubernetes.io/projected/25eb2ba5-e303-4a08-8720-1b60361c2d5e-kube-api-access-t4vnh\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.352623 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.352569 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.352623 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.352599 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.352709 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.352623 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.352709 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.352653 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.352709 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.352689 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.352908 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.352717 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.352908 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.352749 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.352908 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.352824 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.353247 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.353223 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.353393 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.353352 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.353617 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.353591 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.353749 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.353600 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.353955 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.353788 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.355443 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.355418 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.355891 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.355870 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.409812 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.409739 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.410121 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.410102 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4vnh\" (UniqueName: \"kubernetes.io/projected/25eb2ba5-e303-4a08-8720-1b60361c2d5e-kube-api-access-t4vnh\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.461182 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.461157 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:30.893819 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.893792 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb"] Nov 01 07:05:30.895054 ip-10-0-140-68 kubenswrapper[4385]: W1101 07:05:30.895028 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25eb2ba5_e303_4a08_8720_1b60361c2d5e.slice/crio-7292109b2a734e681350999af2a8e4c6874e739035b27e0b1f16a59ce8122fa9 WatchSource:0}: Error finding container 7292109b2a734e681350999af2a8e4c6874e739035b27e0b1f16a59ce8122fa9: Status 404 returned error can't find the container with id 7292109b2a734e681350999af2a8e4c6874e739035b27e0b1f16a59ce8122fa9 Nov 01 07:05:30.897754 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.897719 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:05:30.897863 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.897799 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:05:30.897924 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:30.897858 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:05:31.907119 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:31.907084 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" event={"ID":"25eb2ba5-e303-4a08-8720-1b60361c2d5e","Type":"ContainerStarted","Data":"ac87782f32e0492fca094f01742a66d50f683bdd139b1dba85762049d29ca52a"} Nov 01 07:05:31.907614 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:31.907146 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" event={"ID":"25eb2ba5-e303-4a08-8720-1b60361c2d5e","Type":"ContainerStarted","Data":"7292109b2a734e681350999af2a8e4c6874e739035b27e0b1f16a59ce8122fa9"} Nov 01 07:05:32.090178 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:32.083433 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" podStartSLOduration=2.0834137249999998 podStartE2EDuration="2.083413725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:05:32.083343727 +0000 UTC m=+3341.847143585" watchObservedRunningTime="2026-11-01 07:05:32.083413725 +0000 UTC m=+3341.847213583" Nov 01 07:05:32.461902 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:32.461871 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:32.463579 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:32.463558 4385 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 01 07:05:32.463642 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:32.463623 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" podUID="25eb2ba5-e303-4a08-8720-1b60361c2d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 01 07:05:33.414753 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:33.414717 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:05:33.461542 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:33.461510 4385 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 01 07:05:33.461720 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:33.461567 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" podUID="25eb2ba5-e303-4a08-8720-1b60361c2d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 01 07:05:34.462064 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:34.462019 4385 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 01 07:05:34.462545 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:34.462088 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" podUID="25eb2ba5-e303-4a08-8720-1b60361c2d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 01 07:05:35.028414 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:35.028373 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:05:35.194861 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:35.192248 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:05:35.461898 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:35.461863 4385 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 01 07:05:35.462087 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:35.461913 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" podUID="25eb2ba5-e303-4a08-8720-1b60361c2d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 01 07:05:36.462508 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:36.462471 4385 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 01 07:05:36.462977 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:36.462534 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" podUID="25eb2ba5-e303-4a08-8720-1b60361c2d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 01 07:05:36.812875 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:05:36.812774 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:05:37.466485 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:37.466455 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:37.467013 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:37.466849 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:37.467487 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:37.467467 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:05:37.787444 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:37.787345 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:05:38.267785 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:38.267725 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:05:42.954880 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:42.948577 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:05:45.263489 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.263453 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg"] Nov 01 07:05:45.263999 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.263985 4385 cpu_manager.go:414] "containerMap: removing container" podUID="becaf42b-35ef-46db-a4f8-0cc8780722f0" containerName="cleanup" Nov 01 07:05:45.264053 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.264000 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="becaf42b-35ef-46db-a4f8-0cc8780722f0" containerName="cleanup" Nov 01 07:05:45.264053 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.264013 4385 cpu_manager.go:414] "containerMap: removing container" podUID="f3b1e9b1-6c86-4818-bab1-715cad35fb8c" containerName="cleanup" Nov 01 07:05:45.264053 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.264018 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b1e9b1-6c86-4818-bab1-715cad35fb8c" containerName="cleanup" Nov 01 07:05:45.264053 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.264035 4385 cpu_manager.go:414] "containerMap: removing container" podUID="247e6488-ee32-4931-9670-ea759c4cc896" containerName="cleanup" Nov 01 07:05:45.264053 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.264040 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247e6488-ee32-4931-9670-ea759c4cc896" containerName="cleanup" Nov 01 07:05:45.264053 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.264048 4385 cpu_manager.go:414] "containerMap: removing container" podUID="9debfa95-4fbc-4962-8177-6a53821135ca" containerName="cleanup" Nov 01 07:05:45.264053 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.264052 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9debfa95-4fbc-4962-8177-6a53821135ca" containerName="cleanup" Nov 01 07:05:45.264276 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.264066 4385 cpu_manager.go:414] "containerMap: removing container" podUID="60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac" containerName="cleanup" Nov 01 07:05:45.264276 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.264071 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac" containerName="cleanup" Nov 01 07:05:45.264276 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.264084 4385 cpu_manager.go:414] "containerMap: removing container" podUID="3871d20b-6d9f-4c6e-87b1-8e66cc4ef350" containerName="cleanup" Nov 01 07:05:45.264276 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.264089 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3871d20b-6d9f-4c6e-87b1-8e66cc4ef350" containerName="cleanup" Nov 01 07:05:45.264276 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.264175 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="247e6488-ee32-4931-9670-ea759c4cc896" containerName="cleanup" Nov 01 07:05:45.264276 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.264189 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="d27e5671-8155-4605-9333-001d14195f62" containerName="cleanup" Nov 01 07:05:45.264276 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.264197 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b6bd30-e1f3-4e9f-b7be-ecde9cfad8ac" containerName="cleanup" Nov 01 07:05:45.264276 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.264203 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="3871d20b-6d9f-4c6e-87b1-8e66cc4ef350" containerName="cleanup" Nov 01 07:05:45.268076 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.268043 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.378958 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.378916 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.379128 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.379065 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.379128 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.379111 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.379270 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.379248 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba7312af-84e4-4e52-b6ff-239c756ff8af-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.379341 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.379326 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.379417 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.379399 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.379483 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.379467 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.379568 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.379553 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.379664 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.379580 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvj7d\" (UniqueName: \"kubernetes.io/projected/ba7312af-84e4-4e52-b6ff-239c756ff8af-kube-api-access-xvj7d\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.471264 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.471223 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg"] Nov 01 07:05:45.481166 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.481131 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba7312af-84e4-4e52-b6ff-239c756ff8af-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.481732 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.481238 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.481912 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.481774 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.481912 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.481864 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.482070 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.481931 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.482070 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.481956 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvj7d\" (UniqueName: \"kubernetes.io/projected/ba7312af-84e4-4e52-b6ff-239c756ff8af-kube-api-access-xvj7d\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.482070 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.481973 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.482070 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.481999 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.482305 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.482090 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.482305 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.482118 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.482426 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.482321 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.483676 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.483612 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.484050 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.484007 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.485175 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.485149 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.486333 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.485160 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba7312af-84e4-4e52-b6ff-239c756ff8af-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.493465 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.493437 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.538983 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.538924 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.539678 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.539660 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvj7d\" (UniqueName: \"kubernetes.io/projected/ba7312af-84e4-4e52-b6ff-239c756ff8af-kube-api-access-xvj7d\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:45.586092 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:45.586062 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:46.019982 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:46.019948 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg"] Nov 01 07:05:46.023381 ip-10-0-140-68 kubenswrapper[4385]: W1101 07:05:46.023355 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba7312af_84e4_4e52_b6ff_239c756ff8af.slice/crio-0f9e188d10552d8610b6d7ca48c7489d95f23fdb88682b3b21fbde45b97d8bd6 WatchSource:0}: Error finding container 0f9e188d10552d8610b6d7ca48c7489d95f23fdb88682b3b21fbde45b97d8bd6: Status 404 returned error can't find the container with id 0f9e188d10552d8610b6d7ca48c7489d95f23fdb88682b3b21fbde45b97d8bd6 Nov 01 07:05:46.026362 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:46.026334 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:05:46.026452 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:46.026392 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:05:46.026452 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:46.026433 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:05:47.019036 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:47.018998 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" event={"ID":"ba7312af-84e4-4e52-b6ff-239c756ff8af","Type":"ContainerStarted","Data":"dde515518b6933248e18f788f817c78a7b555db0edc68e15c0b873e8e63bad10"} Nov 01 07:05:47.019518 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:47.019055 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" event={"ID":"ba7312af-84e4-4e52-b6ff-239c756ff8af","Type":"ContainerStarted","Data":"0f9e188d10552d8610b6d7ca48c7489d95f23fdb88682b3b21fbde45b97d8bd6"} Nov 01 07:05:47.245940 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:47.245884 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" podStartSLOduration=2.245870026 podStartE2EDuration="2.245870026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:05:47.193308869 +0000 UTC m=+3356.957108727" watchObservedRunningTime="2026-11-01 07:05:47.245870026 +0000 UTC m=+3357.009669882" Nov 01 07:05:47.592806 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:47.592772 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:47.593217 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:47.593193 4385 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 01 07:05:47.593400 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:47.593378 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" podUID="ba7312af-84e4-4e52-b6ff-239c756ff8af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 01 07:05:47.639589 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:47.639557 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:05:48.587053 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:48.586588 4385 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 01 07:05:48.587053 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:48.586650 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" podUID="ba7312af-84e4-4e52-b6ff-239c756ff8af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 01 07:05:49.592874 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:49.592445 4385 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 01 07:05:49.592874 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:49.592516 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" podUID="ba7312af-84e4-4e52-b6ff-239c756ff8af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 01 07:05:50.599523 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:50.599487 4385 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 01 07:05:50.600275 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:50.600244 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" podUID="ba7312af-84e4-4e52-b6ff-239c756ff8af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 01 07:05:51.595859 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:51.595146 4385 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 01 07:05:51.595859 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:51.595222 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" podUID="ba7312af-84e4-4e52-b6ff-239c756ff8af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 01 07:05:52.422328 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:52.422292 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:05:52.688478 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:52.688387 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:52.688982 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:52.688962 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:52.691456 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:52.691429 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:05:52.802334 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:52.802291 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:05:57.048814 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:05:57.048778 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:00.384637 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.384598 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h"] Nov 01 07:06:00.389186 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.389163 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.486125 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.486094 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h"] Nov 01 07:06:00.564539 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.564500 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.564722 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.564568 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/815320bb-e707-4f6b-8df8-3956fc925e51-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.564816 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.564797 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.564917 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.564901 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/815320bb-e707-4f6b-8df8-3956fc925e51-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.565029 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.565011 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56k87\" (UniqueName: \"kubernetes.io/projected/815320bb-e707-4f6b-8df8-3956fc925e51-kube-api-access-56k87\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.565073 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.565042 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.565116 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.565104 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/815320bb-e707-4f6b-8df8-3956fc925e51-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.565161 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.565146 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.565215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.565204 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.666147 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.666068 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/815320bb-e707-4f6b-8df8-3956fc925e51-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.666147 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.666104 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.666385 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.666190 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/815320bb-e707-4f6b-8df8-3956fc925e51-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.666503 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.666377 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56k87\" (UniqueName: \"kubernetes.io/projected/815320bb-e707-4f6b-8df8-3956fc925e51-kube-api-access-56k87\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.666503 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.666425 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.666886 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.666515 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/815320bb-e707-4f6b-8df8-3956fc925e51-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.666886 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.666549 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.666886 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.666585 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.666886 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.666622 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.666886 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.666687 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.666886 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.666782 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.667277 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.667244 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.667405 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.667385 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.667761 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.667737 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/815320bb-e707-4f6b-8df8-3956fc925e51-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.669043 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.669023 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/815320bb-e707-4f6b-8df8-3956fc925e51-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.669302 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.669282 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.722418 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.722388 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/815320bb-e707-4f6b-8df8-3956fc925e51-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:00.722788 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:00.722770 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56k87\" (UniqueName: \"kubernetes.io/projected/815320bb-e707-4f6b-8df8-3956fc925e51-kube-api-access-56k87\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:01.002740 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:01.002638 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:01.507315 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:01.507285 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h"] Nov 01 07:06:01.521774 ip-10-0-140-68 kubenswrapper[4385]: W1101 07:06:01.520040 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815320bb_e707_4f6b_8df8_3956fc925e51.slice/crio-6fd40df698bb7bd34898f0212453ea1ea115dfe9ee86a5af68cf566ad78da324 WatchSource:0}: Error finding container 6fd40df698bb7bd34898f0212453ea1ea115dfe9ee86a5af68cf566ad78da324: Status 404 returned error can't find the container with id 6fd40df698bb7bd34898f0212453ea1ea115dfe9ee86a5af68cf566ad78da324 Nov 01 07:06:01.524354 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:01.522909 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:06:01.524354 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:01.522994 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:06:01.524354 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:01.523040 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:06:01.736677 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:01.736641 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:02.146273 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:02.146238 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" event={"ID":"815320bb-e707-4f6b-8df8-3956fc925e51","Type":"ContainerStarted","Data":"fa4c2b24bea7e6d049b86f5e70bc4205594f00198558c130b48350a4b34c51c4"} Nov 01 07:06:02.146476 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:02.146299 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" event={"ID":"815320bb-e707-4f6b-8df8-3956fc925e51","Type":"ContainerStarted","Data":"6fd40df698bb7bd34898f0212453ea1ea115dfe9ee86a5af68cf566ad78da324"} Nov 01 07:06:02.316555 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:02.316481 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" podStartSLOduration=2.31646825 podStartE2EDuration="2.31646825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:06:02.313265203 +0000 UTC m=+3372.077065076" watchObservedRunningTime="2026-11-01 07:06:02.31646825 +0000 UTC m=+3372.080268109" Nov 01 07:06:03.003933 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:03.003889 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:03.005579 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:03.005553 4385 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 01 07:06:03.005706 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:03.005599 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" podUID="815320bb-e707-4f6b-8df8-3956fc925e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 01 07:06:04.003531 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:04.003499 4385 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 01 07:06:04.003709 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:04.003549 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" podUID="815320bb-e707-4f6b-8df8-3956fc925e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 01 07:06:05.003850 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:05.003802 4385 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 01 07:06:05.004296 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:05.003873 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" podUID="815320bb-e707-4f6b-8df8-3956fc925e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 01 07:06:06.003641 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:06.003602 4385 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 01 07:06:06.003872 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:06.003680 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" podUID="815320bb-e707-4f6b-8df8-3956fc925e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 01 07:06:06.762041 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:06.762000 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:06.808950 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:06:06.808911 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:06:07.003177 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:07.003137 4385 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 01 07:06:07.003365 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:07.003216 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" podUID="815320bb-e707-4f6b-8df8-3956fc925e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 01 07:06:08.191702 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:08.191663 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:08.192248 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:08.192222 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:08.193594 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:08.193572 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:06:10.492653 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:10.492618 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:11.424045 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:11.424010 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:12.905395 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:12.905363 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b"] Nov 01 07:06:12.909570 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:12.909549 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:12.961891 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:12.961863 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-3ba2f3-openshift-default-dockercfg-nrd89\"" Nov 01 07:06:13.017900 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.017869 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b"] Nov 01 07:06:13.021637 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.021598 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.021910 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.021647 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8v7k\" (UniqueName: \"kubernetes.io/projected/ceea7f64-b7d4-4954-a474-9c80d954cab9-kube-api-access-v8v7k\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.021910 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.021717 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.021910 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.021787 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.021910 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.021885 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceea7f64-b7d4-4954-a474-9c80d954cab9-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.022090 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.022026 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.022126 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.022085 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.022164 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.022150 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.022214 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.022197 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.122998 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.122958 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.123197 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.123017 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.123197 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.123072 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.123197 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.123112 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.123197 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.123186 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.123435 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.123216 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8v7k\" (UniqueName: \"kubernetes.io/projected/ceea7f64-b7d4-4954-a474-9c80d954cab9-kube-api-access-v8v7k\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.123435 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.123246 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.123549 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.123440 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.123549 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.123476 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.123549 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.123504 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.123549 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.123529 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceea7f64-b7d4-4954-a474-9c80d954cab9-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.123778 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.123558 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.123878 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.123801 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.124104 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.124085 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceea7f64-b7d4-4954-a474-9c80d954cab9-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.125648 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.125626 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.125738 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.125704 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.179547 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.179487 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.179672 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.179652 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8v7k\" (UniqueName: \"kubernetes.io/projected/ceea7f64-b7d4-4954-a474-9c80d954cab9-kube-api-access-v8v7k\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.224319 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.224295 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:13.657367 ip-10-0-140-68 kubenswrapper[4385]: W1101 07:06:13.657331 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceea7f64_b7d4_4954_a474_9c80d954cab9.slice/crio-a63b9b533209f800435b9ba571fff143d1431f801763858a70b2272578856db2 WatchSource:0}: Error finding container a63b9b533209f800435b9ba571fff143d1431f801763858a70b2272578856db2: Status 404 returned error can't find the container with id a63b9b533209f800435b9ba571fff143d1431f801763858a70b2272578856db2 Nov 01 07:06:13.657874 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.657855 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b"] Nov 01 07:06:13.659695 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.659663 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:06:13.659778 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.659724 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:06:13.659778 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:13.659753 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:06:14.256179 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:14.256139 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" event={"ID":"ceea7f64-b7d4-4954-a474-9c80d954cab9","Type":"ContainerStarted","Data":"8768329706d1936d36ccebe4378cc624156fe5e70412d98edba86177aff7e8ee"} Nov 01 07:06:14.256634 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:14.256194 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" event={"ID":"ceea7f64-b7d4-4954-a474-9c80d954cab9","Type":"ContainerStarted","Data":"a63b9b533209f800435b9ba571fff143d1431f801763858a70b2272578856db2"} Nov 01 07:06:14.398417 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:14.398363 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" podStartSLOduration=2.39834829 podStartE2EDuration="2.39834829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:06:14.369570064 +0000 UTC m=+3384.133369922" watchObservedRunningTime="2026-11-01 07:06:14.39834829 +0000 UTC m=+3384.162148220" Nov 01 07:06:15.224649 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:15.224614 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:15.226485 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:15.226456 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 07:06:15.226618 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:15.226507 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" podUID="ceea7f64-b7d4-4954-a474-9c80d954cab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 07:06:15.534290 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:15.534206 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:16.225548 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:16.225512 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 07:06:16.225747 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:16.225575 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" podUID="ceea7f64-b7d4-4954-a474-9c80d954cab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 07:06:16.254408 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:16.254375 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:17.225277 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:17.225243 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 07:06:17.225685 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:17.225300 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" podUID="ceea7f64-b7d4-4954-a474-9c80d954cab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 07:06:18.225484 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:18.225448 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 07:06:18.225930 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:18.225499 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" podUID="ceea7f64-b7d4-4954-a474-9c80d954cab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 07:06:19.224766 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:19.224732 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 07:06:19.224965 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:19.224793 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" podUID="ceea7f64-b7d4-4954-a474-9c80d954cab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 07:06:20.225209 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:20.225172 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 07:06:20.225664 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:20.225239 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" podUID="ceea7f64-b7d4-4954-a474-9c80d954cab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 07:06:21.029368 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:21.029332 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:21.230787 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:21.230755 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:21.231304 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:21.231151 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:21.231941 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:21.231921 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:06:25.680167 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:25.680111 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:27.059579 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:27.059542 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:29.624463 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:29.624253 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:29.772848 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:29.772799 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:29.919220 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:29.919135 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:30.426478 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:30.426443 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:33.983787 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:33.983752 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:34.126576 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:34.126544 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:35.128915 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:35.128879 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:36.780636 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:06:36.780590 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:06:39.830000 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:39.829969 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:42.612978 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:42.612943 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:42.791822 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:42.791782 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:43.321197 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:43.321165 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:44.565183 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:44.565140 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:44.747144 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:44.747107 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:49.100181 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:49.100133 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:49.348426 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:49.348382 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:49.555892 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:49.555769 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:51.686884 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:51.686819 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:52.298682 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:52.298648 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:53.137526 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:53.137489 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:53.932990 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:53.932953 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:55.666786 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:55.666744 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:55.817945 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:55.817908 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:55.967621 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:55.967542 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:57.668361 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:57.668333 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v"] Nov 01 07:06:57.777058 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:57.777024 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr"] Nov 01 07:06:58.039658 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:58.039575 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:58.188851 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:58.188804 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:58.339571 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:58.339541 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:58.351771 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:58.351742 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh"] Nov 01 07:06:58.351989 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:58.351966 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" podUID="a800ad90-b099-4484-bbf5-0c1ce2802c73" containerName="maas-api" containerID="cri-o://0b6349d6c8c247088c84765d0b0d7cd3b5758fccd53eb11cafc09622ff6fc0b2" gracePeriod=30 Nov 01 07:06:58.360683 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:58.360649 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" podUID="a800ad90-b099-4484-bbf5-0c1ce2802c73" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.106:8443/health\": dial tcp 10.134.0.106:8443: connect: connection refused" Nov 01 07:06:58.522334 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:58.522301 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:58.684148 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:58.684040 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:58.878605 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:58.878571 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:59.026148 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:59.026057 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:59.609273 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:59.604297 4385 generic.go:356] "Generic (PLEG): container finished" podID="a800ad90-b099-4484-bbf5-0c1ce2802c73" containerID="0b6349d6c8c247088c84765d0b0d7cd3b5758fccd53eb11cafc09622ff6fc0b2" exitCode=0 Nov 01 07:06:59.609273 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:59.604506 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" event={"ID":"a800ad90-b099-4484-bbf5-0c1ce2802c73","Type":"ContainerDied","Data":"0b6349d6c8c247088c84765d0b0d7cd3b5758fccd53eb11cafc09622ff6fc0b2"} Nov 01 07:06:59.660657 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:59.660626 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:06:59.675888 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:59.675803 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" Nov 01 07:06:59.879461 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:59.879417 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a800ad90-b099-4484-bbf5-0c1ce2802c73-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a800ad90-b099-4484-bbf5-0c1ce2802c73-maas-api-tls\") pod \"a800ad90-b099-4484-bbf5-0c1ce2802c73\" (UID: \"a800ad90-b099-4484-bbf5-0c1ce2802c73\") " Nov 01 07:06:59.880008 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:59.879554 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a800ad90-b099-4484-bbf5-0c1ce2802c73-kube-api-access-s5ngk\" (UniqueName: \"kubernetes.io/projected/a800ad90-b099-4484-bbf5-0c1ce2802c73-kube-api-access-s5ngk\") pod \"a800ad90-b099-4484-bbf5-0c1ce2802c73\" (UID: \"a800ad90-b099-4484-bbf5-0c1ce2802c73\") " Nov 01 07:06:59.880008 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:59.879589 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a800ad90-b099-4484-bbf5-0c1ce2802c73-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a800ad90-b099-4484-bbf5-0c1ce2802c73-openshift-service-ca\") pod \"a800ad90-b099-4484-bbf5-0c1ce2802c73\" (UID: \"a800ad90-b099-4484-bbf5-0c1ce2802c73\") " Nov 01 07:06:59.880008 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:59.879654 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a800ad90-b099-4484-bbf5-0c1ce2802c73-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a800ad90-b099-4484-bbf5-0c1ce2802c73-metrics-certs\") pod \"a800ad90-b099-4484-bbf5-0c1ce2802c73\" (UID: \"a800ad90-b099-4484-bbf5-0c1ce2802c73\") " Nov 01 07:06:59.881254 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:59.881190 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a800ad90-b099-4484-bbf5-0c1ce2802c73-openshift-service-ca" pod "a800ad90-b099-4484-bbf5-0c1ce2802c73" (UID: "a800ad90-b099-4484-bbf5-0c1ce2802c73"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:06:59.883253 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:59.883227 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a800ad90-b099-4484-bbf5-0c1ce2802c73-metrics-certs" pod "a800ad90-b099-4484-bbf5-0c1ce2802c73" (UID: "a800ad90-b099-4484-bbf5-0c1ce2802c73"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:06:59.883685 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:59.883664 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a800ad90-b099-4484-bbf5-0c1ce2802c73-kube-api-access-s5ngk" pod "a800ad90-b099-4484-bbf5-0c1ce2802c73" (UID: "a800ad90-b099-4484-bbf5-0c1ce2802c73"). InnerVolumeSpecName "kube-api-access-s5ngk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:06:59.885286 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:59.885262 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a800ad90-b099-4484-bbf5-0c1ce2802c73-maas-api-tls" pod "a800ad90-b099-4484-bbf5-0c1ce2802c73" (UID: "a800ad90-b099-4484-bbf5-0c1ce2802c73"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:06:59.981061 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:59.980978 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a800ad90-b099-4484-bbf5-0c1ce2802c73-maas-api-tls\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:06:59.981061 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:59.981012 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5ngk\" (UniqueName: \"kubernetes.io/projected/a800ad90-b099-4484-bbf5-0c1ce2802c73-kube-api-access-s5ngk\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:06:59.981061 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:59.981027 4385 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a800ad90-b099-4484-bbf5-0c1ce2802c73-openshift-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:06:59.981061 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:06:59.981041 4385 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a800ad90-b099-4484-bbf5-0c1ce2802c73-metrics-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:00.614327 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:00.614295 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" Nov 01 07:07:00.614579 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:00.614325 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh" event={"ID":"a800ad90-b099-4484-bbf5-0c1ce2802c73","Type":"ContainerDied","Data":"847e8a0a64c1b21e5f477e9e1bc3a40588be18c31b6bb21e760a0e8ee58b6377"} Nov 01 07:07:00.614579 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:00.614399 4385 scope.go:122] "RemoveContainer" containerID="0b6349d6c8c247088c84765d0b0d7cd3b5758fccd53eb11cafc09622ff6fc0b2" Nov 01 07:07:00.789175 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:00.789094 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh"] Nov 01 07:07:00.838454 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:00.838425 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-25719d-7f77668659-dw9rh"] Nov 01 07:07:01.145900 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:01.145864 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:01.332264 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:01.332224 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:01.590617 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:01.590580 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:01.830463 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:01.830433 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:01.982598 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:01.982507 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:02.386244 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:02.386211 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a800ad90-b099-4484-bbf5-0c1ce2802c73" path="/var/lib/kubelet/pods/a800ad90-b099-4484-bbf5-0c1ce2802c73/volumes" Nov 01 07:07:02.669625 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:02.669493 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v" podUID="20dfe3dd-6dab-4021-827a-eed61d2846e7" containerName="payload-processing" containerID="cri-o://3ce7b00e13433f3aa9a72745f7764156871b7716e41333533f16854dcc6a7feb" gracePeriod=25 Nov 01 07:07:02.777378 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:02.777345 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr" podUID="43d636e4-561d-4a2f-905f-5b864155bb8c" containerName="payload-pre-processing" containerID="cri-o://bf00a0e4935f5301839909779b8f845d66e7a3258f75d62484d54e4090596285" gracePeriod=25 Nov 01 07:07:03.806111 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:03.806079 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:04.625674 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:04.625633 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:04.766854 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:04.766072 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:04.884446 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:04.884345 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr" podUID="43d636e4-561d-4a2f-905f-5b864155bb8c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:07:04.884446 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:04.884408 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v" podUID="20dfe3dd-6dab-4021-827a-eed61d2846e7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:07:06.740571 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:06.740533 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:06.797776 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:07:06.797729 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:07:07.612676 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:07.612641 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:07.762959 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:07.762922 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:09.170134 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:09.170102 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:09.506483 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:09.506368 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:09.712190 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:09.712157 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:09.903274 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:09.903241 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:10.467868 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:10.467821 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:10.618562 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:10.618521 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:11.025097 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:11.025052 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:11.526534 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:11.526499 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h"] Nov 01 07:07:11.527017 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:11.526698 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" podUID="815320bb-e707-4f6b-8df8-3956fc925e51" containerName="istio-proxy" containerID="cri-o://fa4c2b24bea7e6d049b86f5e70bc4205594f00198558c130b48350a4b34c51c4" gracePeriod=30 Nov 01 07:07:11.538033 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:11.537993 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:11.539482 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:11.539455 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg"] Nov 01 07:07:11.539720 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:11.539682 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" podUID="ba7312af-84e4-4e52-b6ff-239c756ff8af" containerName="istio-proxy" containerID="cri-o://dde515518b6933248e18f788f817c78a7b555db0edc68e15c0b873e8e63bad10" gracePeriod=30 Nov 01 07:07:11.562490 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:11.562461 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb"] Nov 01 07:07:11.562687 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:11.562668 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" podUID="25eb2ba5-e303-4a08-8720-1b60361c2d5e" containerName="istio-proxy" containerID="cri-o://ac87782f32e0492fca094f01742a66d50f683bdd139b1dba85762049d29ca52a" gracePeriod=30 Nov 01 07:07:11.865950 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:11.865917 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:14.882687 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:14.882652 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v" podUID="20dfe3dd-6dab-4021-827a-eed61d2846e7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:07:14.883114 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:14.882653 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr" podUID="43d636e4-561d-4a2f-905f-5b864155bb8c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:07:15.499399 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:15.499365 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:15.698901 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:15.698818 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:16.763808 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.763774 4385 generic.go:356] "Generic (PLEG): container finished" podID="ba7312af-84e4-4e52-b6ff-239c756ff8af" containerID="dde515518b6933248e18f788f817c78a7b555db0edc68e15c0b873e8e63bad10" exitCode=0 Nov 01 07:07:16.764283 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.763894 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" event={"ID":"ba7312af-84e4-4e52-b6ff-239c756ff8af","Type":"ContainerDied","Data":"dde515518b6933248e18f788f817c78a7b555db0edc68e15c0b873e8e63bad10"} Nov 01 07:07:16.765892 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.765864 4385 generic.go:356] "Generic (PLEG): container finished" podID="815320bb-e707-4f6b-8df8-3956fc925e51" containerID="fa4c2b24bea7e6d049b86f5e70bc4205594f00198558c130b48350a4b34c51c4" exitCode=0 Nov 01 07:07:16.766123 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.765865 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" event={"ID":"815320bb-e707-4f6b-8df8-3956fc925e51","Type":"ContainerDied","Data":"fa4c2b24bea7e6d049b86f5e70bc4205594f00198558c130b48350a4b34c51c4"} Nov 01 07:07:16.767944 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.767923 4385 generic.go:356] "Generic (PLEG): container finished" podID="25eb2ba5-e303-4a08-8720-1b60361c2d5e" containerID="ac87782f32e0492fca094f01742a66d50f683bdd139b1dba85762049d29ca52a" exitCode=0 Nov 01 07:07:16.768049 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.768033 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" event={"ID":"25eb2ba5-e303-4a08-8720-1b60361c2d5e","Type":"ContainerDied","Data":"ac87782f32e0492fca094f01742a66d50f683bdd139b1dba85762049d29ca52a"} Nov 01 07:07:16.826430 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.826398 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:16.844014 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.843967 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:07:16.884593 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.884563 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:07:16.904749 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.904714 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:07:16.945859 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.945353 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-credential-socket\") pod \"815320bb-e707-4f6b-8df8-3956fc925e51\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " Nov 01 07:07:16.945859 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.945412 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-credential-socket\") pod \"ba7312af-84e4-4e52-b6ff-239c756ff8af\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " Nov 01 07:07:16.945859 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.945461 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba7312af-84e4-4e52-b6ff-239c756ff8af-kube-api-access-xvj7d\" (UniqueName: \"kubernetes.io/projected/ba7312af-84e4-4e52-b6ff-239c756ff8af-kube-api-access-xvj7d\") pod \"ba7312af-84e4-4e52-b6ff-239c756ff8af\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " Nov 01 07:07:16.945859 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.945496 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-token\" (UniqueName: \"kubernetes.io/projected/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-token\") pod \"ba7312af-84e4-4e52-b6ff-239c756ff8af\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " Nov 01 07:07:16.945859 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.945533 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/815320bb-e707-4f6b-8df8-3956fc925e51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/815320bb-e707-4f6b-8df8-3956fc925e51-istio-podinfo\") pod \"815320bb-e707-4f6b-8df8-3956fc925e51\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " Nov 01 07:07:16.945859 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.945565 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-envoy\") pod \"ba7312af-84e4-4e52-b6ff-239c756ff8af\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " Nov 01 07:07:16.945859 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.945595 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba7312af-84e4-4e52-b6ff-239c756ff8af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba7312af-84e4-4e52-b6ff-239c756ff8af-istiod-ca-cert\") pod \"ba7312af-84e4-4e52-b6ff-239c756ff8af\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " Nov 01 07:07:16.945859 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.945627 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25eb2ba5-e303-4a08-8720-1b60361c2d5e-kube-api-access-t4vnh\" (UniqueName: \"kubernetes.io/projected/25eb2ba5-e303-4a08-8720-1b60361c2d5e-kube-api-access-t4vnh\") pod \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " Nov 01 07:07:16.945859 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.945655 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/815320bb-e707-4f6b-8df8-3956fc925e51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/815320bb-e707-4f6b-8df8-3956fc925e51-istiod-ca-cert\") pod \"815320bb-e707-4f6b-8df8-3956fc925e51\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " Nov 01 07:07:16.945859 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.945711 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-workload-certs\") pod \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " Nov 01 07:07:16.945859 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.945742 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istiod-ca-cert\") pod \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " Nov 01 07:07:16.945859 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.945780 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/815320bb-e707-4f6b-8df8-3956fc925e51-istio-token\" (UniqueName: \"kubernetes.io/projected/815320bb-e707-4f6b-8df8-3956fc925e51-istio-token\") pod \"815320bb-e707-4f6b-8df8-3956fc925e51\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " Nov 01 07:07:16.946661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.945886 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-istio-envoy\") pod \"815320bb-e707-4f6b-8df8-3956fc925e51\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " Nov 01 07:07:16.946661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.945932 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-podinfo\") pod \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " Nov 01 07:07:16.946661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.945968 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-istio-data\") pod \"815320bb-e707-4f6b-8df8-3956fc925e51\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " Nov 01 07:07:16.946661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.946050 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-envoy\") pod \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " Nov 01 07:07:16.946661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.946085 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-data\") pod \"ba7312af-84e4-4e52-b6ff-239c756ff8af\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " Nov 01 07:07:16.946661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.946159 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-data\") pod \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " Nov 01 07:07:16.946661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.946163 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-credential-socket" pod "815320bb-e707-4f6b-8df8-3956fc925e51" (UID: "815320bb-e707-4f6b-8df8-3956fc925e51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:16.946661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.946189 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-workload-certs\") pod \"815320bb-e707-4f6b-8df8-3956fc925e51\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " Nov 01 07:07:16.946661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.946232 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-workload-certs\") pod \"ba7312af-84e4-4e52-b6ff-239c756ff8af\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " Nov 01 07:07:16.946661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.946284 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/815320bb-e707-4f6b-8df8-3956fc925e51-kube-api-access-56k87\" (UniqueName: \"kubernetes.io/projected/815320bb-e707-4f6b-8df8-3956fc925e51-kube-api-access-56k87\") pod \"815320bb-e707-4f6b-8df8-3956fc925e51\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " Nov 01 07:07:16.946661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.946315 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-token\" (UniqueName: \"kubernetes.io/projected/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-token\") pod \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " Nov 01 07:07:16.946661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.946343 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-podinfo\") pod \"ba7312af-84e4-4e52-b6ff-239c756ff8af\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " Nov 01 07:07:16.946661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.946380 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-workload-socket\") pod \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " Nov 01 07:07:16.946661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.946403 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-credential-socket" pod "ba7312af-84e4-4e52-b6ff-239c756ff8af" (UID: "ba7312af-84e4-4e52-b6ff-239c756ff8af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:16.946661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.946422 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-credential-socket\") pod \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\" (UID: \"25eb2ba5-e303-4a08-8720-1b60361c2d5e\") " Nov 01 07:07:16.946661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.946452 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-workload-socket\") pod \"ba7312af-84e4-4e52-b6ff-239c756ff8af\" (UID: \"ba7312af-84e4-4e52-b6ff-239c756ff8af\") " Nov 01 07:07:16.946661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.946548 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-workload-socket\") pod \"815320bb-e707-4f6b-8df8-3956fc925e51\" (UID: \"815320bb-e707-4f6b-8df8-3956fc925e51\") " Nov 01 07:07:16.947689 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.947036 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:16.947689 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.947056 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:16.947689 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.947330 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-workload-socket" pod "815320bb-e707-4f6b-8df8-3956fc925e51" (UID: "815320bb-e707-4f6b-8df8-3956fc925e51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:16.947689 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.947522 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-istio-data" pod "815320bb-e707-4f6b-8df8-3956fc925e51" (UID: "815320bb-e707-4f6b-8df8-3956fc925e51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:16.950683 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.950651 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-envoy" pod "25eb2ba5-e303-4a08-8720-1b60361c2d5e" (UID: "25eb2ba5-e303-4a08-8720-1b60361c2d5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:16.950919 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.950900 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-data" pod "ba7312af-84e4-4e52-b6ff-239c756ff8af" (UID: "ba7312af-84e4-4e52-b6ff-239c756ff8af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:16.951101 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.951083 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-data" pod "25eb2ba5-e303-4a08-8720-1b60361c2d5e" (UID: "25eb2ba5-e303-4a08-8720-1b60361c2d5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:16.951277 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.951260 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-workload-certs" pod "815320bb-e707-4f6b-8df8-3956fc925e51" (UID: "815320bb-e707-4f6b-8df8-3956fc925e51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:16.951444 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.951426 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-workload-certs" pod "ba7312af-84e4-4e52-b6ff-239c756ff8af" (UID: "ba7312af-84e4-4e52-b6ff-239c756ff8af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:16.952698 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.952666 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-credential-socket" pod "25eb2ba5-e303-4a08-8720-1b60361c2d5e" (UID: "25eb2ba5-e303-4a08-8720-1b60361c2d5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:16.952896 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.952871 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-workload-socket" pod "25eb2ba5-e303-4a08-8720-1b60361c2d5e" (UID: "25eb2ba5-e303-4a08-8720-1b60361c2d5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:16.953499 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.953252 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/815320bb-e707-4f6b-8df8-3956fc925e51-istiod-ca-cert" pod "815320bb-e707-4f6b-8df8-3956fc925e51" (UID: "815320bb-e707-4f6b-8df8-3956fc925e51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:16.953978 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.953644 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-workload-socket" pod "ba7312af-84e4-4e52-b6ff-239c756ff8af" (UID: "ba7312af-84e4-4e52-b6ff-239c756ff8af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:16.954308 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.954249 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istiod-ca-cert" pod "25eb2ba5-e303-4a08-8720-1b60361c2d5e" (UID: "25eb2ba5-e303-4a08-8720-1b60361c2d5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:16.954499 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.954481 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-workload-certs" pod "25eb2ba5-e303-4a08-8720-1b60361c2d5e" (UID: "25eb2ba5-e303-4a08-8720-1b60361c2d5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:16.956580 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.956558 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba7312af-84e4-4e52-b6ff-239c756ff8af-istiod-ca-cert" pod "ba7312af-84e4-4e52-b6ff-239c756ff8af" (UID: "ba7312af-84e4-4e52-b6ff-239c756ff8af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:16.957173 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.957145 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/815320bb-e707-4f6b-8df8-3956fc925e51-kube-api-access-56k87" pod "815320bb-e707-4f6b-8df8-3956fc925e51" (UID: "815320bb-e707-4f6b-8df8-3956fc925e51"). InnerVolumeSpecName "kube-api-access-56k87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:16.966316 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.966280 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-envoy" pod "ba7312af-84e4-4e52-b6ff-239c756ff8af" (UID: "ba7312af-84e4-4e52-b6ff-239c756ff8af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:16.966455 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.966366 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/815320bb-e707-4f6b-8df8-3956fc925e51-istio-podinfo" pod "815320bb-e707-4f6b-8df8-3956fc925e51" (UID: "815320bb-e707-4f6b-8df8-3956fc925e51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:07:16.966523 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.966456 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7312af-84e4-4e52-b6ff-239c756ff8af-kube-api-access-xvj7d" pod "ba7312af-84e4-4e52-b6ff-239c756ff8af" (UID: "ba7312af-84e4-4e52-b6ff-239c756ff8af"). InnerVolumeSpecName "kube-api-access-xvj7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:16.966922 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.966883 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-podinfo" pod "ba7312af-84e4-4e52-b6ff-239c756ff8af" (UID: "ba7312af-84e4-4e52-b6ff-239c756ff8af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:07:16.978949 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.970478 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-token" pod "ba7312af-84e4-4e52-b6ff-239c756ff8af" (UID: "ba7312af-84e4-4e52-b6ff-239c756ff8af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:16.978949 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.970551 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-podinfo" pod "25eb2ba5-e303-4a08-8720-1b60361c2d5e" (UID: "25eb2ba5-e303-4a08-8720-1b60361c2d5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:07:16.978949 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.970699 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-token" pod "25eb2ba5-e303-4a08-8720-1b60361c2d5e" (UID: "25eb2ba5-e303-4a08-8720-1b60361c2d5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:16.978949 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.970791 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25eb2ba5-e303-4a08-8720-1b60361c2d5e-kube-api-access-t4vnh" pod "25eb2ba5-e303-4a08-8720-1b60361c2d5e" (UID: "25eb2ba5-e303-4a08-8720-1b60361c2d5e"). InnerVolumeSpecName "kube-api-access-t4vnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:16.978949 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.971252 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/815320bb-e707-4f6b-8df8-3956fc925e51-istio-token" pod "815320bb-e707-4f6b-8df8-3956fc925e51" (UID: "815320bb-e707-4f6b-8df8-3956fc925e51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:16.981422 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:16.980005 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-istio-envoy" pod "815320bb-e707-4f6b-8df8-3956fc925e51" (UID: "815320bb-e707-4f6b-8df8-3956fc925e51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:17.048236 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048200 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.048236 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048237 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.048539 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048263 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.048539 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048280 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.048539 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048294 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.048539 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048306 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.048539 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048327 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.048539 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048341 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.048539 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048356 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56k87\" (UniqueName: \"kubernetes.io/projected/815320bb-e707-4f6b-8df8-3956fc925e51-kube-api-access-56k87\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.048539 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048369 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.048539 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048384 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.048539 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048404 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.048539 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048419 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.048539 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048432 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.048539 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048483 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/815320bb-e707-4f6b-8df8-3956fc925e51-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.048539 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048498 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvj7d\" (UniqueName: \"kubernetes.io/projected/ba7312af-84e4-4e52-b6ff-239c756ff8af-kube-api-access-xvj7d\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.048539 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048509 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.048539 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048529 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/815320bb-e707-4f6b-8df8-3956fc925e51-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.048539 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048543 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba7312af-84e4-4e52-b6ff-239c756ff8af-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.049425 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048557 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba7312af-84e4-4e52-b6ff-239c756ff8af-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.049425 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048570 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4vnh\" (UniqueName: \"kubernetes.io/projected/25eb2ba5-e303-4a08-8720-1b60361c2d5e-kube-api-access-t4vnh\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.049425 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048590 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/815320bb-e707-4f6b-8df8-3956fc925e51-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.049425 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048602 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25eb2ba5-e303-4a08-8720-1b60361c2d5e-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.049425 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048616 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25eb2ba5-e303-4a08-8720-1b60361c2d5e-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.049425 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.048644 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/815320bb-e707-4f6b-8df8-3956fc925e51-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.125587 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.125534 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:17.778323 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.778296 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" Nov 01 07:07:17.778779 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.778331 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb" event={"ID":"25eb2ba5-e303-4a08-8720-1b60361c2d5e","Type":"ContainerDied","Data":"7292109b2a734e681350999af2a8e4c6874e739035b27e0b1f16a59ce8122fa9"} Nov 01 07:07:17.778779 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.778401 4385 scope.go:122] "RemoveContainer" containerID="ac87782f32e0492fca094f01742a66d50f683bdd139b1dba85762049d29ca52a" Nov 01 07:07:17.780464 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.780438 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" Nov 01 07:07:17.780580 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.780471 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg" event={"ID":"ba7312af-84e4-4e52-b6ff-239c756ff8af","Type":"ContainerDied","Data":"0f9e188d10552d8610b6d7ca48c7489d95f23fdb88682b3b21fbde45b97d8bd6"} Nov 01 07:07:17.788374 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.788351 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" event={"ID":"815320bb-e707-4f6b-8df8-3956fc925e51","Type":"ContainerDied","Data":"6fd40df698bb7bd34898f0212453ea1ea115dfe9ee86a5af68cf566ad78da324"} Nov 01 07:07:17.788495 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.788361 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h" Nov 01 07:07:17.792667 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.792648 4385 scope.go:122] "RemoveContainer" containerID="dde515518b6933248e18f788f817c78a7b555db0edc68e15c0b873e8e63bad10" Nov 01 07:07:17.806986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.806967 4385 scope.go:122] "RemoveContainer" containerID="fa4c2b24bea7e6d049b86f5e70bc4205594f00198558c130b48350a4b34c51c4" Nov 01 07:07:17.883596 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.883571 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw"] Nov 01 07:07:17.993454 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.993414 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h"] Nov 01 07:07:17.996846 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:17.996812 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-28g7h"] Nov 01 07:07:18.105184 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:18.105151 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb"] Nov 01 07:07:18.109583 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:18.109553 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-k28nb"] Nov 01 07:07:18.219089 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:18.219006 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg"] Nov 01 07:07:18.231166 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:18.231138 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-jcsgg"] Nov 01 07:07:18.239022 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:18.238995 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:18.384305 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:18.384278 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25eb2ba5-e303-4a08-8720-1b60361c2d5e" path="/var/lib/kubelet/pods/25eb2ba5-e303-4a08-8720-1b60361c2d5e/volumes" Nov 01 07:07:18.384824 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:18.384789 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:18.385016 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:18.384999 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="815320bb-e707-4f6b-8df8-3956fc925e51" path="/var/lib/kubelet/pods/815320bb-e707-4f6b-8df8-3956fc925e51/volumes" Nov 01 07:07:18.385465 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:18.385452 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba7312af-84e4-4e52-b6ff-239c756ff8af" path="/var/lib/kubelet/pods/ba7312af-84e4-4e52-b6ff-239c756ff8af/volumes" Nov 01 07:07:21.708823 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:21.708789 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:22.731284 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:22.731248 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:22.826194 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:22.826164 4385 generic.go:356] "Generic (PLEG): container finished" podID="20dfe3dd-6dab-4021-827a-eed61d2846e7" containerID="3ce7b00e13433f3aa9a72745f7764156871b7716e41333533f16854dcc6a7feb" exitCode=0 Nov 01 07:07:22.826348 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:22.826244 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v" event={"ID":"20dfe3dd-6dab-4021-827a-eed61d2846e7","Type":"ContainerDied","Data":"3ce7b00e13433f3aa9a72745f7764156871b7716e41333533f16854dcc6a7feb"} Nov 01 07:07:22.884223 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:22.884183 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw" podUID="0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b" containerName="payload-processing" containerID="cri-o://b718bc68779a3e98187f9a7c837808bfe4981f59025760137bafa47d7ba4beb5" gracePeriod=25 Nov 01 07:07:22.927952 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:22.927884 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v" Nov 01 07:07:23.014590 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.014563 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20dfe3dd-6dab-4021-827a-eed61d2846e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20dfe3dd-6dab-4021-827a-eed61d2846e7-plugins-config-volume\") pod \"20dfe3dd-6dab-4021-827a-eed61d2846e7\" (UID: \"20dfe3dd-6dab-4021-827a-eed61d2846e7\") " Nov 01 07:07:23.014733 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.014720 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20dfe3dd-6dab-4021-827a-eed61d2846e7-kube-api-access-dc6zl\" (UniqueName: \"kubernetes.io/projected/20dfe3dd-6dab-4021-827a-eed61d2846e7-kube-api-access-dc6zl\") pod \"20dfe3dd-6dab-4021-827a-eed61d2846e7\" (UID: \"20dfe3dd-6dab-4021-827a-eed61d2846e7\") " Nov 01 07:07:23.014925 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.014905 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20dfe3dd-6dab-4021-827a-eed61d2846e7-plugins-config-volume" pod "20dfe3dd-6dab-4021-827a-eed61d2846e7" (UID: "20dfe3dd-6dab-4021-827a-eed61d2846e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:23.015078 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.015065 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20dfe3dd-6dab-4021-827a-eed61d2846e7-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:23.016862 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.016814 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20dfe3dd-6dab-4021-827a-eed61d2846e7-kube-api-access-dc6zl" pod "20dfe3dd-6dab-4021-827a-eed61d2846e7" (UID: "20dfe3dd-6dab-4021-827a-eed61d2846e7"). InnerVolumeSpecName "kube-api-access-dc6zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:23.029997 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.029979 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr" Nov 01 07:07:23.116127 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.116096 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d636e4-561d-4a2f-905f-5b864155bb8c-kube-api-access-82qxs\" (UniqueName: \"kubernetes.io/projected/43d636e4-561d-4a2f-905f-5b864155bb8c-kube-api-access-82qxs\") pod \"43d636e4-561d-4a2f-905f-5b864155bb8c\" (UID: \"43d636e4-561d-4a2f-905f-5b864155bb8c\") " Nov 01 07:07:23.116294 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.116277 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43d636e4-561d-4a2f-905f-5b864155bb8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d636e4-561d-4a2f-905f-5b864155bb8c-plugins-config-volume\") pod \"43d636e4-561d-4a2f-905f-5b864155bb8c\" (UID: \"43d636e4-561d-4a2f-905f-5b864155bb8c\") " Nov 01 07:07:23.116645 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.116621 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d636e4-561d-4a2f-905f-5b864155bb8c-plugins-config-volume" pod "43d636e4-561d-4a2f-905f-5b864155bb8c" (UID: "43d636e4-561d-4a2f-905f-5b864155bb8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:23.116749 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.116714 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc6zl\" (UniqueName: \"kubernetes.io/projected/20dfe3dd-6dab-4021-827a-eed61d2846e7-kube-api-access-dc6zl\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:23.116749 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.116731 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d636e4-561d-4a2f-905f-5b864155bb8c-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:23.118356 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.118340 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d636e4-561d-4a2f-905f-5b864155bb8c-kube-api-access-82qxs" pod "43d636e4-561d-4a2f-905f-5b864155bb8c" (UID: "43d636e4-561d-4a2f-905f-5b864155bb8c"). InnerVolumeSpecName "kube-api-access-82qxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:23.218235 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.218161 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82qxs\" (UniqueName: \"kubernetes.io/projected/43d636e4-561d-4a2f-905f-5b864155bb8c-kube-api-access-82qxs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:23.833638 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.833606 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v" event={"ID":"20dfe3dd-6dab-4021-827a-eed61d2846e7","Type":"ContainerDied","Data":"8429ff03c9d39388896c6c37f4ea81ed116de32de6e7e83fcaaa4a9259f33a41"} Nov 01 07:07:23.833638 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.833628 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v" Nov 01 07:07:23.834168 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.833668 4385 scope.go:122] "RemoveContainer" containerID="3ce7b00e13433f3aa9a72745f7764156871b7716e41333533f16854dcc6a7feb" Nov 01 07:07:23.835184 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.835161 4385 generic.go:356] "Generic (PLEG): container finished" podID="43d636e4-561d-4a2f-905f-5b864155bb8c" containerID="bf00a0e4935f5301839909779b8f845d66e7a3258f75d62484d54e4090596285" exitCode=0 Nov 01 07:07:23.835333 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.835233 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr" Nov 01 07:07:23.835333 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.835244 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr" event={"ID":"43d636e4-561d-4a2f-905f-5b864155bb8c","Type":"ContainerDied","Data":"bf00a0e4935f5301839909779b8f845d66e7a3258f75d62484d54e4090596285"} Nov 01 07:07:23.835333 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.835294 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr" event={"ID":"43d636e4-561d-4a2f-905f-5b864155bb8c","Type":"ContainerDied","Data":"463f9ef304455dd640e70c9e5930116ef619df1c3ad51120397f380978f1f43e"} Nov 01 07:07:23.844309 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.844282 4385 scope.go:122] "RemoveContainer" containerID="bf00a0e4935f5301839909779b8f845d66e7a3258f75d62484d54e4090596285" Nov 01 07:07:23.853439 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.853419 4385 scope.go:122] "RemoveContainer" containerID="bf00a0e4935f5301839909779b8f845d66e7a3258f75d62484d54e4090596285" Nov 01 07:07:23.853697 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:07:23.853682 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf00a0e4935f5301839909779b8f845d66e7a3258f75d62484d54e4090596285\": container with ID starting with bf00a0e4935f5301839909779b8f845d66e7a3258f75d62484d54e4090596285 not found: ID does not exist" containerID="bf00a0e4935f5301839909779b8f845d66e7a3258f75d62484d54e4090596285" Nov 01 07:07:23.853750 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.853704 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf00a0e4935f5301839909779b8f845d66e7a3258f75d62484d54e4090596285"} err="failed to get container status \"bf00a0e4935f5301839909779b8f845d66e7a3258f75d62484d54e4090596285\": rpc error: code = NotFound desc = could not find container \"bf00a0e4935f5301839909779b8f845d66e7a3258f75d62484d54e4090596285\": container with ID starting with bf00a0e4935f5301839909779b8f845d66e7a3258f75d62484d54e4090596285 not found: ID does not exist" Nov 01 07:07:23.997583 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:23.997494 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr"] Nov 01 07:07:24.047149 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:24.047115 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-25719d-66ffb897b7-9n4pr"] Nov 01 07:07:24.052187 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:24.052153 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw" podUID="0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:07:24.109650 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:24.109620 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v"] Nov 01 07:07:24.112728 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:24.112703 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-25719d-78689cc99f-2kl4v"] Nov 01 07:07:24.385204 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:24.385171 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20dfe3dd-6dab-4021-827a-eed61d2846e7" path="/var/lib/kubelet/pods/20dfe3dd-6dab-4021-827a-eed61d2846e7/volumes" Nov 01 07:07:24.385583 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:24.385569 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d636e4-561d-4a2f-905f-5b864155bb8c" path="/var/lib/kubelet/pods/43d636e4-561d-4a2f-905f-5b864155bb8c/volumes" Nov 01 07:07:26.489336 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:26.489304 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:28.531533 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:28.531496 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:28.676571 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:28.676535 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:29.103009 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:29.102972 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:29.255430 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:29.255388 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:29.403524 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:29.403427 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:30.415755 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:30.415719 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:30.606120 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:30.604742 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:30.775947 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:30.775819 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:31.248913 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:31.248883 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:31.406208 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:31.406130 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:34.052658 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:34.052613 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw" podUID="0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:07:35.990292 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:35.990252 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:36.791617 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:07:36.791558 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:07:37.702550 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:37.702493 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:37.861980 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:37.861947 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:40.622650 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:40.622619 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:43.143246 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:43.143182 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw" Nov 01 07:07:43.214730 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:43.214695 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b-kube-api-access-kkjc5\" (UniqueName: \"kubernetes.io/projected/0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b-kube-api-access-kkjc5\") pod \"0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b\" (UID: \"0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b\") " Nov 01 07:07:43.215017 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:43.214806 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b-plugins-config-volume\") pod \"0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b\" (UID: \"0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b\") " Nov 01 07:07:43.215260 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:43.215237 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b-plugins-config-volume" pod "0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b" (UID: "0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:43.217138 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:43.217117 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b-kube-api-access-kkjc5" pod "0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b" (UID: "0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b"). InnerVolumeSpecName "kube-api-access-kkjc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:43.316581 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:43.316550 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkjc5\" (UniqueName: \"kubernetes.io/projected/0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b-kube-api-access-kkjc5\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:43.316581 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:43.316577 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:43.744116 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:43.744082 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:43.957332 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:43.957299 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:43.984443 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:43.984409 4385 generic.go:356] "Generic (PLEG): container finished" podID="0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b" containerID="b718bc68779a3e98187f9a7c837808bfe4981f59025760137bafa47d7ba4beb5" exitCode=0 Nov 01 07:07:43.984611 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:43.984464 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw" Nov 01 07:07:43.984611 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:43.984496 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw" event={"ID":"0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b","Type":"ContainerDied","Data":"b718bc68779a3e98187f9a7c837808bfe4981f59025760137bafa47d7ba4beb5"} Nov 01 07:07:43.984611 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:43.984554 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw" event={"ID":"0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b","Type":"ContainerDied","Data":"c26028f275879a7f4d8a31ecafbb17721cfd3dd9f0dd8c1ccf37f0531c51662b"} Nov 01 07:07:43.984611 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:43.984583 4385 scope.go:122] "RemoveContainer" containerID="b718bc68779a3e98187f9a7c837808bfe4981f59025760137bafa47d7ba4beb5" Nov 01 07:07:43.995015 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:43.994986 4385 scope.go:122] "RemoveContainer" containerID="b718bc68779a3e98187f9a7c837808bfe4981f59025760137bafa47d7ba4beb5" Nov 01 07:07:43.995288 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:07:43.995268 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b718bc68779a3e98187f9a7c837808bfe4981f59025760137bafa47d7ba4beb5\": container with ID starting with b718bc68779a3e98187f9a7c837808bfe4981f59025760137bafa47d7ba4beb5 not found: ID does not exist" containerID="b718bc68779a3e98187f9a7c837808bfe4981f59025760137bafa47d7ba4beb5" Nov 01 07:07:43.995348 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:43.995298 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b718bc68779a3e98187f9a7c837808bfe4981f59025760137bafa47d7ba4beb5"} err="failed to get container status \"b718bc68779a3e98187f9a7c837808bfe4981f59025760137bafa47d7ba4beb5\": rpc error: code = NotFound desc = could not find container \"b718bc68779a3e98187f9a7c837808bfe4981f59025760137bafa47d7ba4beb5\": container with ID starting with b718bc68779a3e98187f9a7c837808bfe4981f59025760137bafa47d7ba4beb5 not found: ID does not exist" Nov 01 07:07:44.149585 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:44.149551 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw"] Nov 01 07:07:44.198911 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:44.198883 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1ff1dc-7fc785c587-zk2dw"] Nov 01 07:07:44.199731 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:44.199701 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:44.384264 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:44.384232 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b" path="/var/lib/kubelet/pods/0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b/volumes" Nov 01 07:07:44.774098 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:44.773994 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:46.515294 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:46.515252 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:48.122283 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:48.122250 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:50.197682 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:50.197646 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:50.772508 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:50.772474 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:50.978282 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:50.978245 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:51.219133 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:51.219098 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:51.364285 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:51.364245 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:52.273091 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:52.272993 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:52.801982 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:52.801948 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:53.475646 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:53.475602 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:53.631088 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:53.631056 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:54.133185 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:54.133143 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:55.787820 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:55.787782 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:56.506069 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:56.506015 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:07:59.852665 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:07:59.852631 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:00.189106 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:00.189013 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5"] Nov 01 07:08:00.561909 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:00.559474 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:00.796856 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:00.796332 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:00.869875 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:00.860219 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9"] Nov 01 07:08:00.869875 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:00.860476 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" podUID="0b9a9dca-7256-4ff9-9bab-dc46d8198aba" containerName="maas-api" containerID="cri-o://8ec5d170b1449f41b0b76dfdfc3bf109c18e9e3846d4a8e5d430af27eec68ad2" gracePeriod=30 Nov 01 07:08:01.004522 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:01.004489 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:01.078989 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:01.078943 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" podUID="0b9a9dca-7256-4ff9-9bab-dc46d8198aba" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.131:8443/health\": dial tcp 10.134.0.131:8443: connect: connection refused" Nov 01 07:08:01.118690 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:01.118570 4385 generic.go:356] "Generic (PLEG): container finished" podID="0b9a9dca-7256-4ff9-9bab-dc46d8198aba" containerID="8ec5d170b1449f41b0b76dfdfc3bf109c18e9e3846d4a8e5d430af27eec68ad2" exitCode=0 Nov 01 07:08:01.118690 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:01.118628 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" event={"ID":"0b9a9dca-7256-4ff9-9bab-dc46d8198aba","Type":"ContainerDied","Data":"8ec5d170b1449f41b0b76dfdfc3bf109c18e9e3846d4a8e5d430af27eec68ad2"} Nov 01 07:08:01.270698 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:01.270669 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" Nov 01 07:08:01.396498 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:01.396390 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-metrics-certs\") pod \"0b9a9dca-7256-4ff9-9bab-dc46d8198aba\" (UID: \"0b9a9dca-7256-4ff9-9bab-dc46d8198aba\") " Nov 01 07:08:01.396498 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:01.396487 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-maas-api-tls\") pod \"0b9a9dca-7256-4ff9-9bab-dc46d8198aba\" (UID: \"0b9a9dca-7256-4ff9-9bab-dc46d8198aba\") " Nov 01 07:08:01.396765 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:01.396523 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-openshift-service-ca\") pod \"0b9a9dca-7256-4ff9-9bab-dc46d8198aba\" (UID: \"0b9a9dca-7256-4ff9-9bab-dc46d8198aba\") " Nov 01 07:08:01.396765 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:01.396569 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-kube-api-access-jc25z\" (UniqueName: \"kubernetes.io/projected/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-kube-api-access-jc25z\") pod \"0b9a9dca-7256-4ff9-9bab-dc46d8198aba\" (UID: \"0b9a9dca-7256-4ff9-9bab-dc46d8198aba\") " Nov 01 07:08:01.397358 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:01.397330 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-openshift-service-ca" pod "0b9a9dca-7256-4ff9-9bab-dc46d8198aba" (UID: "0b9a9dca-7256-4ff9-9bab-dc46d8198aba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:08:01.399360 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:01.399338 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-metrics-certs" pod "0b9a9dca-7256-4ff9-9bab-dc46d8198aba" (UID: "0b9a9dca-7256-4ff9-9bab-dc46d8198aba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:08:01.399605 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:01.399577 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-maas-api-tls" pod "0b9a9dca-7256-4ff9-9bab-dc46d8198aba" (UID: "0b9a9dca-7256-4ff9-9bab-dc46d8198aba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:08:01.399711 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:01.399691 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-kube-api-access-jc25z" pod "0b9a9dca-7256-4ff9-9bab-dc46d8198aba" (UID: "0b9a9dca-7256-4ff9-9bab-dc46d8198aba"). InnerVolumeSpecName "kube-api-access-jc25z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:08:01.498232 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:01.498192 4385 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-metrics-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:01.498232 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:01.498221 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-maas-api-tls\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:01.498232 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:01.498237 4385 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-openshift-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:01.498490 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:01.498250 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc25z\" (UniqueName: \"kubernetes.io/projected/0b9a9dca-7256-4ff9-9bab-dc46d8198aba-kube-api-access-jc25z\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:02.126479 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:02.126448 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" Nov 01 07:08:02.126479 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:02.126467 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9" event={"ID":"0b9a9dca-7256-4ff9-9bab-dc46d8198aba","Type":"ContainerDied","Data":"1f2ae14d7fc1dc05bc59b20f94a3336445b8c70073c082f8823c038b27c82f9a"} Nov 01 07:08:02.127099 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:02.126526 4385 scope.go:122] "RemoveContainer" containerID="8ec5d170b1449f41b0b76dfdfc3bf109c18e9e3846d4a8e5d430af27eec68ad2" Nov 01 07:08:02.289827 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:02.289792 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9"] Nov 01 07:08:02.339206 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:02.339175 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-caf74d-744456f584-8vct9"] Nov 01 07:08:02.385596 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:02.385525 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b9a9dca-7256-4ff9-9bab-dc46d8198aba" path="/var/lib/kubelet/pods/0b9a9dca-7256-4ff9-9bab-dc46d8198aba/volumes" Nov 01 07:08:02.773542 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:02.773455 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:05.190033 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:05.189944 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5" podUID="1a10a1d9-d645-42fe-9aa1-d79a35398d68" containerName="payload-processing" containerID="cri-o://edd9dbc5f7bad91f6439344fec9bdb251df1b6c5f2e21a708eefa065ffbe3c2e" gracePeriod=25 Nov 01 07:08:05.757264 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:05.757227 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:05.769257 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:05.769221 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5" podUID="1a10a1d9-d645-42fe-9aa1-d79a35398d68" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:08:06.782177 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:08:06.782132 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:08:07.318231 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:07.318196 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:09.167004 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:09.166965 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:10.717663 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:10.717626 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:10.917654 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:10.917617 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:11.975096 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:11.975064 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:12.135824 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:12.135790 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:12.289182 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:12.289088 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:12.764184 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:12.764151 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:13.528324 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:13.528287 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:13.720414 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:13.720379 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:13.868642 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:13.868605 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:14.124464 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:14.122528 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:14.175554 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:14.175520 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq"] Nov 01 07:08:14.175744 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:14.175724 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" podUID="3c777717-c462-4ba3-bdc3-7b423808eb9b" containerName="istio-proxy" containerID="cri-o://40b25a0cfa6ca56cc04ba0b78ecaf6b92f2cbd392f062251712074a67230d8aa" gracePeriod=30 Nov 01 07:08:14.279382 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:14.279346 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:14.676960 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:14.676921 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:15.769207 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:15.769175 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5" podUID="1a10a1d9-d645-42fe-9aa1-d79a35398d68" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:08:16.793265 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793226 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9"] Nov 01 07:08:16.793716 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793702 4385 cpu_manager.go:414] "containerMap: removing container" podUID="20dfe3dd-6dab-4021-827a-eed61d2846e7" containerName="payload-processing" Nov 01 07:08:16.793781 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793718 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20dfe3dd-6dab-4021-827a-eed61d2846e7" containerName="payload-processing" Nov 01 07:08:16.793781 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793746 4385 cpu_manager.go:414] "containerMap: removing container" podUID="815320bb-e707-4f6b-8df8-3956fc925e51" containerName="istio-proxy" Nov 01 07:08:16.793781 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793751 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815320bb-e707-4f6b-8df8-3956fc925e51" containerName="istio-proxy" Nov 01 07:08:16.793781 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793758 4385 cpu_manager.go:414] "containerMap: removing container" podUID="0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b" containerName="payload-processing" Nov 01 07:08:16.793781 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793763 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b" containerName="payload-processing" Nov 01 07:08:16.793781 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793772 4385 cpu_manager.go:414] "containerMap: removing container" podUID="ba7312af-84e4-4e52-b6ff-239c756ff8af" containerName="istio-proxy" Nov 01 07:08:16.793781 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793777 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7312af-84e4-4e52-b6ff-239c756ff8af" containerName="istio-proxy" Nov 01 07:08:16.794045 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793787 4385 cpu_manager.go:414] "containerMap: removing container" podUID="a800ad90-b099-4484-bbf5-0c1ce2802c73" containerName="maas-api" Nov 01 07:08:16.794045 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793792 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a800ad90-b099-4484-bbf5-0c1ce2802c73" containerName="maas-api" Nov 01 07:08:16.794045 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793798 4385 cpu_manager.go:414] "containerMap: removing container" podUID="0b9a9dca-7256-4ff9-9bab-dc46d8198aba" containerName="maas-api" Nov 01 07:08:16.794045 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793802 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9a9dca-7256-4ff9-9bab-dc46d8198aba" containerName="maas-api" Nov 01 07:08:16.794045 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793820 4385 cpu_manager.go:414] "containerMap: removing container" podUID="43d636e4-561d-4a2f-905f-5b864155bb8c" containerName="payload-pre-processing" Nov 01 07:08:16.794045 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793825 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d636e4-561d-4a2f-905f-5b864155bb8c" containerName="payload-pre-processing" Nov 01 07:08:16.794045 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793847 4385 cpu_manager.go:414] "containerMap: removing container" podUID="25eb2ba5-e303-4a08-8720-1b60361c2d5e" containerName="istio-proxy" Nov 01 07:08:16.794045 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793852 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25eb2ba5-e303-4a08-8720-1b60361c2d5e" containerName="istio-proxy" Nov 01 07:08:16.794045 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793930 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="20dfe3dd-6dab-4021-827a-eed61d2846e7" containerName="payload-processing" Nov 01 07:08:16.794045 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793943 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dad47cc-7b31-4fd7-ab7a-deaa3cb9e54b" containerName="payload-processing" Nov 01 07:08:16.794045 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793948 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="815320bb-e707-4f6b-8df8-3956fc925e51" containerName="istio-proxy" Nov 01 07:08:16.794045 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793954 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba7312af-84e4-4e52-b6ff-239c756ff8af" containerName="istio-proxy" Nov 01 07:08:16.794045 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793961 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="a800ad90-b099-4484-bbf5-0c1ce2802c73" containerName="maas-api" Nov 01 07:08:16.794045 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793967 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9a9dca-7256-4ff9-9bab-dc46d8198aba" containerName="maas-api" Nov 01 07:08:16.794045 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793974 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d636e4-561d-4a2f-905f-5b864155bb8c" containerName="payload-pre-processing" Nov 01 07:08:16.794045 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.793984 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="25eb2ba5-e303-4a08-8720-1b60361c2d5e" containerName="istio-proxy" Nov 01 07:08:16.799263 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.799242 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.875907 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.875875 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79dv8\" (UniqueName: \"kubernetes.io/projected/b74b50a6-08f2-4bc8-966a-1f23722f2807-kube-api-access-79dv8\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.876127 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.875932 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.876127 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.876044 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74b50a6-08f2-4bc8-966a-1f23722f2807-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.876127 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.876089 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.876127 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.876108 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.876448 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.876220 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.876448 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.876269 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.876448 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.876335 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.876448 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.876360 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.904385 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.904353 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9"] Nov 01 07:08:16.977553 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.977515 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.977747 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.977571 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.977747 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.977607 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.977747 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.977657 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79dv8\" (UniqueName: \"kubernetes.io/projected/b74b50a6-08f2-4bc8-966a-1f23722f2807-kube-api-access-79dv8\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.977747 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.977702 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.977987 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.977778 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74b50a6-08f2-4bc8-966a-1f23722f2807-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.977987 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.977869 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.977987 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.977894 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.977987 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.977957 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.977987 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.977970 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.978259 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.978022 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.978313 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.978300 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.978365 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.978334 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.979059 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.979032 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74b50a6-08f2-4bc8-966a-1f23722f2807-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.980378 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.980355 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:16.980486 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:16.980425 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:17.037018 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:17.036981 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:17.037018 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:17.037014 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79dv8\" (UniqueName: \"kubernetes.io/projected/b74b50a6-08f2-4bc8-966a-1f23722f2807-kube-api-access-79dv8\") pod \"maas-default-gateway-openshift-default-7d9897b54c-8c8g9\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:17.112206 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:17.112169 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:17.542301 ip-10-0-140-68 kubenswrapper[4385]: W1101 07:08:17.542270 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74b50a6_08f2_4bc8_966a_1f23722f2807.slice/crio-a3646831c225ccacaf5e62c0340a8875a49168344b5a6c182a08e0b47a35d05f WatchSource:0}: Error finding container a3646831c225ccacaf5e62c0340a8875a49168344b5a6c182a08e0b47a35d05f: Status 404 returned error can't find the container with id a3646831c225ccacaf5e62c0340a8875a49168344b5a6c182a08e0b47a35d05f Nov 01 07:08:17.542479 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:17.542455 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9"] Nov 01 07:08:17.544678 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:17.544647 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:08:17.544779 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:17.544727 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:08:17.544779 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:17.544770 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:08:18.232113 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:18.232078 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" event={"ID":"b74b50a6-08f2-4bc8-966a-1f23722f2807","Type":"ContainerStarted","Data":"758ad61d7e408860d9388cfdede12373201b5174f4c38d0c7cce905b138f4423"} Nov 01 07:08:18.232650 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:18.232131 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" event={"ID":"b74b50a6-08f2-4bc8-966a-1f23722f2807","Type":"ContainerStarted","Data":"a3646831c225ccacaf5e62c0340a8875a49168344b5a6c182a08e0b47a35d05f"} Nov 01 07:08:18.353999 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:18.353950 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" podStartSLOduration=2.353934911 podStartE2EDuration="2.353934911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:08:18.349967817 +0000 UTC m=+3508.113767679" watchObservedRunningTime="2026-11-01 07:08:18.353934911 +0000 UTC m=+3508.117734822" Nov 01 07:08:19.113394 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.113361 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:19.115328 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.115298 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8c8g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 01 07:08:19.115477 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.115347 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" podUID="b74b50a6-08f2-4bc8-966a-1f23722f2807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 01 07:08:19.146465 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.144815 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt"] Nov 01 07:08:19.416362 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.415645 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:19.554718 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.554650 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:19.559606 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.559584 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 07:08:19.662671 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.662590 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj"] Nov 01 07:08:19.669428 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.669399 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69"] Nov 01 07:08:19.721661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.721628 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:19.725864 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.725264 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-credential-socket\") pod \"3c777717-c462-4ba3-bdc3-7b423808eb9b\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " Nov 01 07:08:19.725864 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.725366 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-podinfo\") pod \"3c777717-c462-4ba3-bdc3-7b423808eb9b\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " Nov 01 07:08:19.725864 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.725422 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-workload-socket\") pod \"3c777717-c462-4ba3-bdc3-7b423808eb9b\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " Nov 01 07:08:19.725864 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.725448 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c777717-c462-4ba3-bdc3-7b423808eb9b-kube-api-access-ttxx9\" (UniqueName: \"kubernetes.io/projected/3c777717-c462-4ba3-bdc3-7b423808eb9b-kube-api-access-ttxx9\") pod \"3c777717-c462-4ba3-bdc3-7b423808eb9b\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " Nov 01 07:08:19.725864 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.725512 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-envoy\") pod \"3c777717-c462-4ba3-bdc3-7b423808eb9b\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " Nov 01 07:08:19.725864 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.725570 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-data\") pod \"3c777717-c462-4ba3-bdc3-7b423808eb9b\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " Nov 01 07:08:19.725864 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.725607 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-workload-certs\") pod \"3c777717-c462-4ba3-bdc3-7b423808eb9b\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " Nov 01 07:08:19.725864 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.725657 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c777717-c462-4ba3-bdc3-7b423808eb9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c777717-c462-4ba3-bdc3-7b423808eb9b-istiod-ca-cert\") pod \"3c777717-c462-4ba3-bdc3-7b423808eb9b\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " Nov 01 07:08:19.725864 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.725687 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-token\" (UniqueName: \"kubernetes.io/projected/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-token\") pod \"3c777717-c462-4ba3-bdc3-7b423808eb9b\" (UID: \"3c777717-c462-4ba3-bdc3-7b423808eb9b\") " Nov 01 07:08:19.730825 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.728020 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-data" pod "3c777717-c462-4ba3-bdc3-7b423808eb9b" (UID: "3c777717-c462-4ba3-bdc3-7b423808eb9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:08:19.730825 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.728798 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-workload-socket" pod "3c777717-c462-4ba3-bdc3-7b423808eb9b" (UID: "3c777717-c462-4ba3-bdc3-7b423808eb9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:08:19.730825 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.729106 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c777717-c462-4ba3-bdc3-7b423808eb9b-istiod-ca-cert" pod "3c777717-c462-4ba3-bdc3-7b423808eb9b" (UID: "3c777717-c462-4ba3-bdc3-7b423808eb9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:08:19.730825 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.730573 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-credential-socket" pod "3c777717-c462-4ba3-bdc3-7b423808eb9b" (UID: "3c777717-c462-4ba3-bdc3-7b423808eb9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:08:19.730825 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.730692 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-podinfo" pod "3c777717-c462-4ba3-bdc3-7b423808eb9b" (UID: "3c777717-c462-4ba3-bdc3-7b423808eb9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:08:19.730825 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.730793 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-workload-certs" pod "3c777717-c462-4ba3-bdc3-7b423808eb9b" (UID: "3c777717-c462-4ba3-bdc3-7b423808eb9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:08:19.731442 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.731412 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-envoy" pod "3c777717-c462-4ba3-bdc3-7b423808eb9b" (UID: "3c777717-c462-4ba3-bdc3-7b423808eb9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:08:19.731710 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.731670 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-token" pod "3c777717-c462-4ba3-bdc3-7b423808eb9b" (UID: "3c777717-c462-4ba3-bdc3-7b423808eb9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:08:19.732869 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.732119 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c777717-c462-4ba3-bdc3-7b423808eb9b-kube-api-access-ttxx9" pod "3c777717-c462-4ba3-bdc3-7b423808eb9b" (UID: "3c777717-c462-4ba3-bdc3-7b423808eb9b"). InnerVolumeSpecName "kube-api-access-ttxx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:08:19.733377 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.733357 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r"] Nov 01 07:08:19.827936 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.827897 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:19.827936 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.827924 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:19.827936 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.827937 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttxx9\" (UniqueName: \"kubernetes.io/projected/3c777717-c462-4ba3-bdc3-7b423808eb9b-kube-api-access-ttxx9\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:19.827936 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.827947 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:19.828242 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.827955 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:19.828242 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.827963 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:19.828242 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.827975 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c777717-c462-4ba3-bdc3-7b423808eb9b-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:19.828242 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.827983 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c777717-c462-4ba3-bdc3-7b423808eb9b-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:19.828242 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.827990 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c777717-c462-4ba3-bdc3-7b423808eb9b-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:19.865281 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:19.865239 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:20.114556 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:20.114521 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8c8g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 01 07:08:20.114719 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:20.114579 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" podUID="b74b50a6-08f2-4bc8-966a-1f23722f2807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 01 07:08:20.247162 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:20.247125 4385 generic.go:356] "Generic (PLEG): container finished" podID="3c777717-c462-4ba3-bdc3-7b423808eb9b" containerID="40b25a0cfa6ca56cc04ba0b78ecaf6b92f2cbd392f062251712074a67230d8aa" exitCode=0 Nov 01 07:08:20.247457 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:20.247437 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" Nov 01 07:08:20.247792 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:20.247771 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" event={"ID":"3c777717-c462-4ba3-bdc3-7b423808eb9b","Type":"ContainerDied","Data":"40b25a0cfa6ca56cc04ba0b78ecaf6b92f2cbd392f062251712074a67230d8aa"} Nov 01 07:08:20.247919 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:20.247818 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq" event={"ID":"3c777717-c462-4ba3-bdc3-7b423808eb9b","Type":"ContainerDied","Data":"c2f0cee76868e0efa1e888a7ec2806fc37167254046534b8959fd5af41b94f73"} Nov 01 07:08:20.247919 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:20.247872 4385 scope.go:122] "RemoveContainer" containerID="40b25a0cfa6ca56cc04ba0b78ecaf6b92f2cbd392f062251712074a67230d8aa" Nov 01 07:08:20.259470 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:20.259448 4385 scope.go:122] "RemoveContainer" containerID="34276e826555c578a861f2d96c0bc61a9b4851851610ecfa40002414df7d95ce" Nov 01 07:08:20.269061 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:20.269036 4385 scope.go:122] "RemoveContainer" containerID="40b25a0cfa6ca56cc04ba0b78ecaf6b92f2cbd392f062251712074a67230d8aa" Nov 01 07:08:20.269398 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:08:20.269380 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40b25a0cfa6ca56cc04ba0b78ecaf6b92f2cbd392f062251712074a67230d8aa\": container with ID starting with 40b25a0cfa6ca56cc04ba0b78ecaf6b92f2cbd392f062251712074a67230d8aa not found: ID does not exist" containerID="40b25a0cfa6ca56cc04ba0b78ecaf6b92f2cbd392f062251712074a67230d8aa" Nov 01 07:08:20.269496 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:20.269406 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40b25a0cfa6ca56cc04ba0b78ecaf6b92f2cbd392f062251712074a67230d8aa"} err="failed to get container status \"40b25a0cfa6ca56cc04ba0b78ecaf6b92f2cbd392f062251712074a67230d8aa\": rpc error: code = NotFound desc = could not find container \"40b25a0cfa6ca56cc04ba0b78ecaf6b92f2cbd392f062251712074a67230d8aa\": container with ID starting with 40b25a0cfa6ca56cc04ba0b78ecaf6b92f2cbd392f062251712074a67230d8aa not found: ID does not exist" Nov 01 07:08:20.269496 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:20.269423 4385 scope.go:122] "RemoveContainer" containerID="34276e826555c578a861f2d96c0bc61a9b4851851610ecfa40002414df7d95ce" Nov 01 07:08:20.269756 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:08:20.269740 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34276e826555c578a861f2d96c0bc61a9b4851851610ecfa40002414df7d95ce\": container with ID starting with 34276e826555c578a861f2d96c0bc61a9b4851851610ecfa40002414df7d95ce not found: ID does not exist" containerID="34276e826555c578a861f2d96c0bc61a9b4851851610ecfa40002414df7d95ce" Nov 01 07:08:20.269812 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:20.269761 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34276e826555c578a861f2d96c0bc61a9b4851851610ecfa40002414df7d95ce"} err="failed to get container status \"34276e826555c578a861f2d96c0bc61a9b4851851610ecfa40002414df7d95ce\": rpc error: code = NotFound desc = could not find container \"34276e826555c578a861f2d96c0bc61a9b4851851610ecfa40002414df7d95ce\": container with ID starting with 34276e826555c578a861f2d96c0bc61a9b4851851610ecfa40002414df7d95ce not found: ID does not exist" Nov 01 07:08:20.412981 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:20.412890 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq"] Nov 01 07:08:20.421234 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:20.421206 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-1ff1dc-openshift-default-6b77ccc689-t9rhq"] Nov 01 07:08:21.113546 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:21.113506 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8c8g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 01 07:08:21.113756 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:21.113589 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" podUID="b74b50a6-08f2-4bc8-966a-1f23722f2807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 01 07:08:21.771109 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:21.771029 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:22.113590 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:22.113545 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8c8g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 01 07:08:22.113812 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:22.113613 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" podUID="b74b50a6-08f2-4bc8-966a-1f23722f2807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 01 07:08:22.416758 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:22.416663 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c777717-c462-4ba3-bdc3-7b423808eb9b" path="/var/lib/kubelet/pods/3c777717-c462-4ba3-bdc3-7b423808eb9b/volumes" Nov 01 07:08:23.113187 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:23.113157 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8c8g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 01 07:08:23.113633 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:23.113608 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" podUID="b74b50a6-08f2-4bc8-966a-1f23722f2807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 01 07:08:24.112727 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:24.112694 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8c8g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 01 07:08:24.112966 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:24.112765 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" podUID="b74b50a6-08f2-4bc8-966a-1f23722f2807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 01 07:08:24.145615 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:24.145575 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt" podUID="94a78d83-1b31-4176-95eb-b3b03b076419" containerName="payload-pre-processing" containerID="cri-o://bf3bc0a112dd2bb69eec360fb7bcc657271fab05cbdc42ccfc1ddd5c1e02eab4" gracePeriod=25 Nov 01 07:08:24.275901 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:24.275855 4385 generic.go:356] "Generic (PLEG): container finished" podID="94a78d83-1b31-4176-95eb-b3b03b076419" containerID="bf3bc0a112dd2bb69eec360fb7bcc657271fab05cbdc42ccfc1ddd5c1e02eab4" exitCode=0 Nov 01 07:08:24.276057 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:24.275927 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt" event={"ID":"94a78d83-1b31-4176-95eb-b3b03b076419","Type":"ContainerDied","Data":"bf3bc0a112dd2bb69eec360fb7bcc657271fab05cbdc42ccfc1ddd5c1e02eab4"} Nov 01 07:08:24.428629 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:24.428605 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt" Nov 01 07:08:24.576678 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:24.576645 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a78d83-1b31-4176-95eb-b3b03b076419-kube-api-access-6wp4c\" (UniqueName: \"kubernetes.io/projected/94a78d83-1b31-4176-95eb-b3b03b076419-kube-api-access-6wp4c\") pod \"94a78d83-1b31-4176-95eb-b3b03b076419\" (UID: \"94a78d83-1b31-4176-95eb-b3b03b076419\") " Nov 01 07:08:24.576865 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:24.576696 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a78d83-1b31-4176-95eb-b3b03b076419-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a78d83-1b31-4176-95eb-b3b03b076419-plugins-config-volume\") pod \"94a78d83-1b31-4176-95eb-b3b03b076419\" (UID: \"94a78d83-1b31-4176-95eb-b3b03b076419\") " Nov 01 07:08:24.577224 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:24.577185 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a78d83-1b31-4176-95eb-b3b03b076419-plugins-config-volume" pod "94a78d83-1b31-4176-95eb-b3b03b076419" (UID: "94a78d83-1b31-4176-95eb-b3b03b076419"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:08:24.579096 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:24.579065 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a78d83-1b31-4176-95eb-b3b03b076419-kube-api-access-6wp4c" pod "94a78d83-1b31-4176-95eb-b3b03b076419" (UID: "94a78d83-1b31-4176-95eb-b3b03b076419"). InnerVolumeSpecName "kube-api-access-6wp4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:08:24.677932 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:24.677828 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wp4c\" (UniqueName: \"kubernetes.io/projected/94a78d83-1b31-4176-95eb-b3b03b076419-kube-api-access-6wp4c\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:24.677932 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:24.677888 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a78d83-1b31-4176-95eb-b3b03b076419-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:24.864139 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:24.864092 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:25.108745 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:25.108702 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:25.112676 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:25.112651 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8c8g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 01 07:08:25.112849 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:25.112707 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" podUID="b74b50a6-08f2-4bc8-966a-1f23722f2807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 01 07:08:25.284703 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:25.284676 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt" Nov 01 07:08:25.285261 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:25.284709 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt" event={"ID":"94a78d83-1b31-4176-95eb-b3b03b076419","Type":"ContainerDied","Data":"9dbbd0963b33a9e54e24852ebf64e323c9f3896b0ca80fd3931b4632b6db48ed"} Nov 01 07:08:25.285261 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:25.284788 4385 scope.go:122] "RemoveContainer" containerID="bf3bc0a112dd2bb69eec360fb7bcc657271fab05cbdc42ccfc1ddd5c1e02eab4" Nov 01 07:08:25.457295 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:25.457182 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt"] Nov 01 07:08:25.476424 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:25.476400 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5" Nov 01 07:08:25.505845 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:25.505798 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-581487-7cb86cd96c-w2dpt"] Nov 01 07:08:25.587425 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:25.587393 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a10a1d9-d645-42fe-9aa1-d79a35398d68-kube-api-access-b77kw\" (UniqueName: \"kubernetes.io/projected/1a10a1d9-d645-42fe-9aa1-d79a35398d68-kube-api-access-b77kw\") pod \"1a10a1d9-d645-42fe-9aa1-d79a35398d68\" (UID: \"1a10a1d9-d645-42fe-9aa1-d79a35398d68\") " Nov 01 07:08:25.587593 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:25.587568 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a10a1d9-d645-42fe-9aa1-d79a35398d68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a10a1d9-d645-42fe-9aa1-d79a35398d68-plugins-config-volume\") pod \"1a10a1d9-d645-42fe-9aa1-d79a35398d68\" (UID: \"1a10a1d9-d645-42fe-9aa1-d79a35398d68\") " Nov 01 07:08:25.588034 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:25.588006 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a10a1d9-d645-42fe-9aa1-d79a35398d68-plugins-config-volume" pod "1a10a1d9-d645-42fe-9aa1-d79a35398d68" (UID: "1a10a1d9-d645-42fe-9aa1-d79a35398d68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:08:25.589919 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:25.589888 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a10a1d9-d645-42fe-9aa1-d79a35398d68-kube-api-access-b77kw" pod "1a10a1d9-d645-42fe-9aa1-d79a35398d68" (UID: "1a10a1d9-d645-42fe-9aa1-d79a35398d68"). InnerVolumeSpecName "kube-api-access-b77kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:08:25.693741 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:25.693658 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a10a1d9-d645-42fe-9aa1-d79a35398d68-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:25.693741 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:25.693696 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b77kw\" (UniqueName: \"kubernetes.io/projected/1a10a1d9-d645-42fe-9aa1-d79a35398d68-kube-api-access-b77kw\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:26.121928 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:26.121898 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:26.122302 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:26.122267 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:26.123120 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:26.123097 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:08:26.293881 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:26.293809 4385 generic.go:356] "Generic (PLEG): container finished" podID="1a10a1d9-d645-42fe-9aa1-d79a35398d68" containerID="edd9dbc5f7bad91f6439344fec9bdb251df1b6c5f2e21a708eefa065ffbe3c2e" exitCode=0 Nov 01 07:08:26.293881 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:26.293872 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5" Nov 01 07:08:26.294464 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:26.293911 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5" event={"ID":"1a10a1d9-d645-42fe-9aa1-d79a35398d68","Type":"ContainerDied","Data":"edd9dbc5f7bad91f6439344fec9bdb251df1b6c5f2e21a708eefa065ffbe3c2e"} Nov 01 07:08:26.294464 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:26.293967 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5" event={"ID":"1a10a1d9-d645-42fe-9aa1-d79a35398d68","Type":"ContainerDied","Data":"181e07b26d66f2ffa7ca696b4a6b0f88812eb9076d428d7468fbf6677bb4f34b"} Nov 01 07:08:26.294464 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:26.293995 4385 scope.go:122] "RemoveContainer" containerID="edd9dbc5f7bad91f6439344fec9bdb251df1b6c5f2e21a708eefa065ffbe3c2e" Nov 01 07:08:26.305455 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:26.305432 4385 scope.go:122] "RemoveContainer" containerID="edd9dbc5f7bad91f6439344fec9bdb251df1b6c5f2e21a708eefa065ffbe3c2e" Nov 01 07:08:26.305734 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:08:26.305717 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edd9dbc5f7bad91f6439344fec9bdb251df1b6c5f2e21a708eefa065ffbe3c2e\": container with ID starting with edd9dbc5f7bad91f6439344fec9bdb251df1b6c5f2e21a708eefa065ffbe3c2e not found: ID does not exist" containerID="edd9dbc5f7bad91f6439344fec9bdb251df1b6c5f2e21a708eefa065ffbe3c2e" Nov 01 07:08:26.305815 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:26.305742 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edd9dbc5f7bad91f6439344fec9bdb251df1b6c5f2e21a708eefa065ffbe3c2e"} err="failed to get container status \"edd9dbc5f7bad91f6439344fec9bdb251df1b6c5f2e21a708eefa065ffbe3c2e\": rpc error: code = NotFound desc = could not find container \"edd9dbc5f7bad91f6439344fec9bdb251df1b6c5f2e21a708eefa065ffbe3c2e\": container with ID starting with edd9dbc5f7bad91f6439344fec9bdb251df1b6c5f2e21a708eefa065ffbe3c2e not found: ID does not exist" Nov 01 07:08:26.401718 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:26.401610 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a78d83-1b31-4176-95eb-b3b03b076419" path="/var/lib/kubelet/pods/94a78d83-1b31-4176-95eb-b3b03b076419/volumes" Nov 01 07:08:26.471913 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:26.471879 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5"] Nov 01 07:08:26.477045 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:26.477018 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6ba6e5-748d8b57c7-n49w5"] Nov 01 07:08:27.844113 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:27.844078 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:28.384178 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:28.384143 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a10a1d9-d645-42fe-9aa1-d79a35398d68" path="/var/lib/kubelet/pods/1a10a1d9-d645-42fe-9aa1-d79a35398d68/volumes" Nov 01 07:08:28.756206 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:28.756117 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:28.949108 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:28.949065 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:29.971163 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:29.971128 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:33.290713 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:33.290672 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:34.732861 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:34.732792 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" podUID="4001c2de-286a-4407-9c74-0cc2492bc8ca" containerName="main" containerID="cri-o://43fcc0d977745f31f2c8a1edd5b8853d912771b841c0808bc59e2f0d335ba79b" gracePeriod=45 Nov 01 07:08:34.748847 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:34.748797 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" podUID="695583a3-4ef8-4cad-ba76-69791506f2f3" containerName="main" containerID="cri-o://836e522c433484c09f0b99c6f55c13812bdfa881d3014e60feb1490f8facd7a7" gracePeriod=45 Nov 01 07:08:34.757299 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:34.757267 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" podUID="f35d3f2c-2ea5-47d6-b808-93a12e4923c0" containerName="main" containerID="cri-o://e1b0edc39f77de1dff156722ed945610c51b04dbbc527d871269363a981eac64" gracePeriod=45 Nov 01 07:08:34.941061 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:34.940879 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:35.037030 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.035192 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 07:08:35.070077 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.070044 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 07:08:35.073479 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.073459 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 07:08:35.204910 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.204880 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-dshm\") pod \"695583a3-4ef8-4cad-ba76-69791506f2f3\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " Nov 01 07:08:35.205098 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.204941 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-kube-api-access-jfmxs\" (UniqueName: \"kubernetes.io/projected/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-kube-api-access-jfmxs\") pod \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " Nov 01 07:08:35.205098 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.204975 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4001c2de-286a-4407-9c74-0cc2492bc8ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/4001c2de-286a-4407-9c74-0cc2492bc8ca-tls-certs\") pod \"4001c2de-286a-4407-9c74-0cc2492bc8ca\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " Nov 01 07:08:35.205098 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.204998 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-home\" (UniqueName: \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-home\") pod \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " Nov 01 07:08:35.205098 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.205020 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-dshm\") pod \"4001c2de-286a-4407-9c74-0cc2492bc8ca\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " Nov 01 07:08:35.205098 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.205041 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4001c2de-286a-4407-9c74-0cc2492bc8ca-kube-api-access-q96zb\" (UniqueName: \"kubernetes.io/projected/4001c2de-286a-4407-9c74-0cc2492bc8ca-kube-api-access-q96zb\") pod \"4001c2de-286a-4407-9c74-0cc2492bc8ca\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " Nov 01 07:08:35.205376 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.205171 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-model-cache\") pod \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " Nov 01 07:08:35.205376 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.205239 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-tls-certs\") pod \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " Nov 01 07:08:35.205376 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.205277 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-tmp-dir\") pod \"4001c2de-286a-4407-9c74-0cc2492bc8ca\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " Nov 01 07:08:35.205376 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.205308 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/695583a3-4ef8-4cad-ba76-69791506f2f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/695583a3-4ef8-4cad-ba76-69791506f2f3-tls-certs\") pod \"695583a3-4ef8-4cad-ba76-69791506f2f3\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " Nov 01 07:08:35.205376 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.205339 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-home" pod "f35d3f2c-2ea5-47d6-b808-93a12e4923c0" (UID: "f35d3f2c-2ea5-47d6-b808-93a12e4923c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:08:35.205376 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.205370 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695583a3-4ef8-4cad-ba76-69791506f2f3-kube-api-access-4v4lf\" (UniqueName: \"kubernetes.io/projected/695583a3-4ef8-4cad-ba76-69791506f2f3-kube-api-access-4v4lf\") pod \"695583a3-4ef8-4cad-ba76-69791506f2f3\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " Nov 01 07:08:35.205718 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.205397 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-tmp-dir\") pod \"695583a3-4ef8-4cad-ba76-69791506f2f3\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " Nov 01 07:08:35.205718 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.205425 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-model-cache\") pod \"695583a3-4ef8-4cad-ba76-69791506f2f3\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " Nov 01 07:08:35.205718 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.205503 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-model-cache\") pod \"4001c2de-286a-4407-9c74-0cc2492bc8ca\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " Nov 01 07:08:35.205718 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.205572 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-tmp-dir\") pod \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " Nov 01 07:08:35.205718 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.205601 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-home\" (UniqueName: \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-home\") pod \"695583a3-4ef8-4cad-ba76-69791506f2f3\" (UID: \"695583a3-4ef8-4cad-ba76-69791506f2f3\") " Nov 01 07:08:35.205718 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.205644 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-home\" (UniqueName: \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-home\") pod \"4001c2de-286a-4407-9c74-0cc2492bc8ca\" (UID: \"4001c2de-286a-4407-9c74-0cc2492bc8ca\") " Nov 01 07:08:35.205718 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.205693 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-dshm\") pod \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\" (UID: \"f35d3f2c-2ea5-47d6-b808-93a12e4923c0\") " Nov 01 07:08:35.207113 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.206375 4385 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-home\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:35.207113 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.206552 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-model-cache" pod "695583a3-4ef8-4cad-ba76-69791506f2f3" (UID: "695583a3-4ef8-4cad-ba76-69791506f2f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:08:35.207113 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.206758 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-tmp-dir" pod "4001c2de-286a-4407-9c74-0cc2492bc8ca" (UID: "4001c2de-286a-4407-9c74-0cc2492bc8ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:08:35.207113 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.207036 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-model-cache" pod "4001c2de-286a-4407-9c74-0cc2492bc8ca" (UID: "4001c2de-286a-4407-9c74-0cc2492bc8ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:08:35.207427 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.207179 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-home" pod "695583a3-4ef8-4cad-ba76-69791506f2f3" (UID: "695583a3-4ef8-4cad-ba76-69791506f2f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:08:35.207427 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.207205 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-home" pod "4001c2de-286a-4407-9c74-0cc2492bc8ca" (UID: "4001c2de-286a-4407-9c74-0cc2492bc8ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:08:35.207549 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.207471 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-tmp-dir" pod "695583a3-4ef8-4cad-ba76-69791506f2f3" (UID: "695583a3-4ef8-4cad-ba76-69791506f2f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:08:35.207549 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.207491 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-model-cache" pod "f35d3f2c-2ea5-47d6-b808-93a12e4923c0" (UID: "f35d3f2c-2ea5-47d6-b808-93a12e4923c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:08:35.207549 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.207522 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-tmp-dir" pod "f35d3f2c-2ea5-47d6-b808-93a12e4923c0" (UID: "f35d3f2c-2ea5-47d6-b808-93a12e4923c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:08:35.208408 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.208371 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-tls-certs" pod "f35d3f2c-2ea5-47d6-b808-93a12e4923c0" (UID: "f35d3f2c-2ea5-47d6-b808-93a12e4923c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:08:35.209303 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.209279 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4001c2de-286a-4407-9c74-0cc2492bc8ca-tls-certs" pod "4001c2de-286a-4407-9c74-0cc2492bc8ca" (UID: "4001c2de-286a-4407-9c74-0cc2492bc8ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:08:35.209459 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.209355 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-kube-api-access-jfmxs" pod "f35d3f2c-2ea5-47d6-b808-93a12e4923c0" (UID: "f35d3f2c-2ea5-47d6-b808-93a12e4923c0"). InnerVolumeSpecName "kube-api-access-jfmxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:08:35.209459 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.209375 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-dshm" pod "4001c2de-286a-4407-9c74-0cc2492bc8ca" (UID: "4001c2de-286a-4407-9c74-0cc2492bc8ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:08:35.210428 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.210405 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-dshm" pod "f35d3f2c-2ea5-47d6-b808-93a12e4923c0" (UID: "f35d3f2c-2ea5-47d6-b808-93a12e4923c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:08:35.210428 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.210419 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/695583a3-4ef8-4cad-ba76-69791506f2f3-tls-certs" pod "695583a3-4ef8-4cad-ba76-69791506f2f3" (UID: "695583a3-4ef8-4cad-ba76-69791506f2f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:08:35.210610 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.210503 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-dshm" pod "695583a3-4ef8-4cad-ba76-69791506f2f3" (UID: "695583a3-4ef8-4cad-ba76-69791506f2f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:08:35.210685 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.210660 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695583a3-4ef8-4cad-ba76-69791506f2f3-kube-api-access-4v4lf" pod "695583a3-4ef8-4cad-ba76-69791506f2f3" (UID: "695583a3-4ef8-4cad-ba76-69791506f2f3"). InnerVolumeSpecName "kube-api-access-4v4lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:08:35.210758 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.210695 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4001c2de-286a-4407-9c74-0cc2492bc8ca-kube-api-access-q96zb" pod "4001c2de-286a-4407-9c74-0cc2492bc8ca" (UID: "4001c2de-286a-4407-9c74-0cc2492bc8ca"). InnerVolumeSpecName "kube-api-access-q96zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:08:35.307493 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.307407 4385 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-tls-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:35.307493 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.307439 4385 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-tmp-dir\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:35.307493 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.307451 4385 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/695583a3-4ef8-4cad-ba76-69791506f2f3-tls-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:35.307493 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.307464 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v4lf\" (UniqueName: \"kubernetes.io/projected/695583a3-4ef8-4cad-ba76-69791506f2f3-kube-api-access-4v4lf\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:35.307493 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.307477 4385 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-tmp-dir\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:35.307493 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.307490 4385 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-model-cache\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:35.307863 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.307502 4385 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-model-cache\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:35.307863 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.307517 4385 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-tmp-dir\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:35.307863 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.307528 4385 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-home\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:35.307863 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.307538 4385 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-home\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:35.307863 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.307548 4385 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-dshm\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:35.307863 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.307563 4385 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/695583a3-4ef8-4cad-ba76-69791506f2f3-dshm\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:35.307863 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.307575 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfmxs\" (UniqueName: \"kubernetes.io/projected/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-kube-api-access-jfmxs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:35.307863 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.307587 4385 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4001c2de-286a-4407-9c74-0cc2492bc8ca-tls-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:35.307863 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.307598 4385 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4001c2de-286a-4407-9c74-0cc2492bc8ca-dshm\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:35.307863 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.307609 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q96zb\" (UniqueName: \"kubernetes.io/projected/4001c2de-286a-4407-9c74-0cc2492bc8ca-kube-api-access-q96zb\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:35.307863 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.307621 4385 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f35d3f2c-2ea5-47d6-b808-93a12e4923c0-model-cache\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:35.358306 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.358268 4385 generic.go:356] "Generic (PLEG): container finished" podID="4001c2de-286a-4407-9c74-0cc2492bc8ca" containerID="43fcc0d977745f31f2c8a1edd5b8853d912771b841c0808bc59e2f0d335ba79b" exitCode=0 Nov 01 07:08:35.358509 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.358315 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" Nov 01 07:08:35.358509 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.358347 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" event={"ID":"4001c2de-286a-4407-9c74-0cc2492bc8ca","Type":"ContainerDied","Data":"43fcc0d977745f31f2c8a1edd5b8853d912771b841c0808bc59e2f0d335ba79b"} Nov 01 07:08:35.358509 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.358389 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj" event={"ID":"4001c2de-286a-4407-9c74-0cc2492bc8ca","Type":"ContainerDied","Data":"9d90ff53a1e74d6f7eda06a766959d010fe94c0026da6e447d156e77ed35d841"} Nov 01 07:08:35.358509 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.358405 4385 scope.go:122] "RemoveContainer" containerID="43fcc0d977745f31f2c8a1edd5b8853d912771b841c0808bc59e2f0d335ba79b" Nov 01 07:08:35.359949 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.359927 4385 generic.go:356] "Generic (PLEG): container finished" podID="f35d3f2c-2ea5-47d6-b808-93a12e4923c0" containerID="e1b0edc39f77de1dff156722ed945610c51b04dbbc527d871269363a981eac64" exitCode=0 Nov 01 07:08:35.360104 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.360034 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" event={"ID":"f35d3f2c-2ea5-47d6-b808-93a12e4923c0","Type":"ContainerDied","Data":"e1b0edc39f77de1dff156722ed945610c51b04dbbc527d871269363a981eac64"} Nov 01 07:08:35.360104 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.360055 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" event={"ID":"f35d3f2c-2ea5-47d6-b808-93a12e4923c0","Type":"ContainerDied","Data":"1774dd11e0e6ef26c9f13d5386f3b96ef04e6903e0f5ac23dc5ff621e016cb3b"} Nov 01 07:08:35.360104 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.360101 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r" Nov 01 07:08:35.366608 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.366584 4385 generic.go:356] "Generic (PLEG): container finished" podID="695583a3-4ef8-4cad-ba76-69791506f2f3" containerID="836e522c433484c09f0b99c6f55c13812bdfa881d3014e60feb1490f8facd7a7" exitCode=0 Nov 01 07:08:35.366745 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.366628 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" Nov 01 07:08:35.366745 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.366666 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" event={"ID":"695583a3-4ef8-4cad-ba76-69791506f2f3","Type":"ContainerDied","Data":"836e522c433484c09f0b99c6f55c13812bdfa881d3014e60feb1490f8facd7a7"} Nov 01 07:08:35.366745 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.366703 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69" event={"ID":"695583a3-4ef8-4cad-ba76-69791506f2f3","Type":"ContainerDied","Data":"8a988f1ad9f22f1346b03cfd3146d01b94a54b010e1437fa096c8caa2c38a22c"} Nov 01 07:08:35.374420 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.374391 4385 scope.go:122] "RemoveContainer" containerID="43fcc0d977745f31f2c8a1edd5b8853d912771b841c0808bc59e2f0d335ba79b" Nov 01 07:08:35.374981 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:08:35.374955 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43fcc0d977745f31f2c8a1edd5b8853d912771b841c0808bc59e2f0d335ba79b\": container with ID starting with 43fcc0d977745f31f2c8a1edd5b8853d912771b841c0808bc59e2f0d335ba79b not found: ID does not exist" containerID="43fcc0d977745f31f2c8a1edd5b8853d912771b841c0808bc59e2f0d335ba79b" Nov 01 07:08:35.375104 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.374994 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43fcc0d977745f31f2c8a1edd5b8853d912771b841c0808bc59e2f0d335ba79b"} err="failed to get container status \"43fcc0d977745f31f2c8a1edd5b8853d912771b841c0808bc59e2f0d335ba79b\": rpc error: code = NotFound desc = could not find container \"43fcc0d977745f31f2c8a1edd5b8853d912771b841c0808bc59e2f0d335ba79b\": container with ID starting with 43fcc0d977745f31f2c8a1edd5b8853d912771b841c0808bc59e2f0d335ba79b not found: ID does not exist" Nov 01 07:08:35.375104 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.375019 4385 scope.go:122] "RemoveContainer" containerID="e1b0edc39f77de1dff156722ed945610c51b04dbbc527d871269363a981eac64" Nov 01 07:08:35.386735 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.386710 4385 scope.go:122] "RemoveContainer" containerID="e1b0edc39f77de1dff156722ed945610c51b04dbbc527d871269363a981eac64" Nov 01 07:08:35.387111 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:08:35.387081 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1b0edc39f77de1dff156722ed945610c51b04dbbc527d871269363a981eac64\": container with ID starting with e1b0edc39f77de1dff156722ed945610c51b04dbbc527d871269363a981eac64 not found: ID does not exist" containerID="e1b0edc39f77de1dff156722ed945610c51b04dbbc527d871269363a981eac64" Nov 01 07:08:35.387232 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.387121 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1b0edc39f77de1dff156722ed945610c51b04dbbc527d871269363a981eac64"} err="failed to get container status \"e1b0edc39f77de1dff156722ed945610c51b04dbbc527d871269363a981eac64\": rpc error: code = NotFound desc = could not find container \"e1b0edc39f77de1dff156722ed945610c51b04dbbc527d871269363a981eac64\": container with ID starting with e1b0edc39f77de1dff156722ed945610c51b04dbbc527d871269363a981eac64 not found: ID does not exist" Nov 01 07:08:35.387232 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.387148 4385 scope.go:122] "RemoveContainer" containerID="836e522c433484c09f0b99c6f55c13812bdfa881d3014e60feb1490f8facd7a7" Nov 01 07:08:35.400021 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.399994 4385 scope.go:122] "RemoveContainer" containerID="836e522c433484c09f0b99c6f55c13812bdfa881d3014e60feb1490f8facd7a7" Nov 01 07:08:35.401201 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:08:35.401173 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836e522c433484c09f0b99c6f55c13812bdfa881d3014e60feb1490f8facd7a7\": container with ID starting with 836e522c433484c09f0b99c6f55c13812bdfa881d3014e60feb1490f8facd7a7 not found: ID does not exist" containerID="836e522c433484c09f0b99c6f55c13812bdfa881d3014e60feb1490f8facd7a7" Nov 01 07:08:35.401330 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.401209 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836e522c433484c09f0b99c6f55c13812bdfa881d3014e60feb1490f8facd7a7"} err="failed to get container status \"836e522c433484c09f0b99c6f55c13812bdfa881d3014e60feb1490f8facd7a7\": rpc error: code = NotFound desc = could not find container \"836e522c433484c09f0b99c6f55c13812bdfa881d3014e60feb1490f8facd7a7\": container with ID starting with 836e522c433484c09f0b99c6f55c13812bdfa881d3014e60feb1490f8facd7a7 not found: ID does not exist" Nov 01 07:08:35.493652 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.493553 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj"] Nov 01 07:08:35.497500 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.497471 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/distinct-2-w2-8ff537-kserve-5d9dfbd7f7-5t5kj"] Nov 01 07:08:35.659203 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.659166 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r"] Nov 01 07:08:35.662765 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.662739 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/simulator-w2-8ff537-kserve-5ddc486ff4-kpv9r"] Nov 01 07:08:35.770672 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.770638 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69"] Nov 01 07:08:35.773547 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:35.773524 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-8ff537/distinct-w2-8ff537-kserve-7b5d9457d-hsf69"] Nov 01 07:08:36.383162 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:36.383126 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4001c2de-286a-4407-9c74-0cc2492bc8ca" path="/var/lib/kubelet/pods/4001c2de-286a-4407-9c74-0cc2492bc8ca/volumes" Nov 01 07:08:36.383552 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:36.383539 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="695583a3-4ef8-4cad-ba76-69791506f2f3" path="/var/lib/kubelet/pods/695583a3-4ef8-4cad-ba76-69791506f2f3/volumes" Nov 01 07:08:36.383949 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:36.383936 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f35d3f2c-2ea5-47d6-b808-93a12e4923c0" path="/var/lib/kubelet/pods/f35d3f2c-2ea5-47d6-b808-93a12e4923c0/volumes" Nov 01 07:08:36.779266 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:08:36.779183 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:08:39.022653 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:39.022615 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:40.724274 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:40.724241 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:44.834977 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:44.834938 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:45.956177 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:45.956142 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:46.952731 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:46.952691 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:47.101334 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:47.101297 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:47.812569 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:47.812529 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:48.022355 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:48.022317 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:48.305930 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:48.305898 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:48.462224 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:48.462189 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:48.618903 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:48.618863 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:49.049785 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:49.049701 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:49.513223 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:49.513185 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:49.962496 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:49.962462 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:50.428636 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:50.428600 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:50.860530 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:50.860494 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:52.938613 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:52.938576 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:53.814453 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:53.814413 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:54.454310 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:54.454279 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2"] Nov 01 07:08:54.503950 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:54.503919 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd"] Nov 01 07:08:54.548608 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:54.548576 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:55.849975 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:55.849872 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:55.958143 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:55.958103 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7"] Nov 01 07:08:56.010060 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:56.010020 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl"] Nov 01 07:08:56.018870 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:56.014954 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx"] Nov 01 07:08:56.051135 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:56.051106 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:56.192054 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:56.191915 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:56.975540 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:56.975503 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:57.127730 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:57.127692 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:57.407684 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:57.407647 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:57.827126 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:57.827094 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:08:59.318398 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:08:59.318366 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:00.430545 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:00.430510 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:00.922066 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:00.922019 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz"] Nov 01 07:09:00.922576 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:00.922531 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" podUID="edc38c4e-77a6-4175-9033-73067cc44525" containerName="istio-proxy" containerID="cri-o://155fc221a054f53da69cdb9d9ae8056c58ad4c78bb1396a7df063c133c17a362" gracePeriod=30 Nov 01 07:09:00.933499 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:00.933444 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:01.359631 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:01.359598 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww"] Nov 01 07:09:01.685112 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:01.685032 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:01.909767 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:01.909737 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:02.648897 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:02.648865 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:02.800380 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:02.800343 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:03.296825 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:03.296788 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:03.447314 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:03.447276 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:05.455303 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:05.455263 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:06.031701 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.031662 4385 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 07:09:06.031882 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.031734 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" podUID="edc38c4e-77a6-4175-9033-73067cc44525" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 07:09:06.213454 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.213389 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 07:09:06.257404 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.257381 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edc38c4e-77a6-4175-9033-73067cc44525-istio-token\" (UniqueName: \"kubernetes.io/projected/edc38c4e-77a6-4175-9033-73067cc44525-istio-token\") pod \"edc38c4e-77a6-4175-9033-73067cc44525\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " Nov 01 07:09:06.257596 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.257415 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edc38c4e-77a6-4175-9033-73067cc44525-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc38c4e-77a6-4175-9033-73067cc44525-istiod-ca-cert\") pod \"edc38c4e-77a6-4175-9033-73067cc44525\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " Nov 01 07:09:06.257596 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.257437 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edc38c4e-77a6-4175-9033-73067cc44525-kube-api-access-qc4l5\" (UniqueName: \"kubernetes.io/projected/edc38c4e-77a6-4175-9033-73067cc44525-kube-api-access-qc4l5\") pod \"edc38c4e-77a6-4175-9033-73067cc44525\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " Nov 01 07:09:06.257596 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.257468 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-istio-envoy\") pod \"edc38c4e-77a6-4175-9033-73067cc44525\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " Nov 01 07:09:06.257596 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.257489 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-istio-data\") pod \"edc38c4e-77a6-4175-9033-73067cc44525\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " Nov 01 07:09:06.257596 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.257508 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-credential-socket\") pod \"edc38c4e-77a6-4175-9033-73067cc44525\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " Nov 01 07:09:06.257913 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.257609 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-workload-certs\") pod \"edc38c4e-77a6-4175-9033-73067cc44525\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " Nov 01 07:09:06.257913 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.257640 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edc38c4e-77a6-4175-9033-73067cc44525-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc38c4e-77a6-4175-9033-73067cc44525-istio-podinfo\") pod \"edc38c4e-77a6-4175-9033-73067cc44525\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " Nov 01 07:09:06.257913 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.257709 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-workload-socket\") pod \"edc38c4e-77a6-4175-9033-73067cc44525\" (UID: \"edc38c4e-77a6-4175-9033-73067cc44525\") " Nov 01 07:09:06.257913 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.257885 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-istio-data" pod "edc38c4e-77a6-4175-9033-73067cc44525" (UID: "edc38c4e-77a6-4175-9033-73067cc44525"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:06.258128 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.257948 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edc38c4e-77a6-4175-9033-73067cc44525-istiod-ca-cert" pod "edc38c4e-77a6-4175-9033-73067cc44525" (UID: "edc38c4e-77a6-4175-9033-73067cc44525"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:09:06.258128 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.258067 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-workload-certs" pod "edc38c4e-77a6-4175-9033-73067cc44525" (UID: "edc38c4e-77a6-4175-9033-73067cc44525"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:06.258128 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.258078 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-workload-socket" pod "edc38c4e-77a6-4175-9033-73067cc44525" (UID: "edc38c4e-77a6-4175-9033-73067cc44525"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:06.258266 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.258196 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-credential-socket" pod "edc38c4e-77a6-4175-9033-73067cc44525" (UID: "edc38c4e-77a6-4175-9033-73067cc44525"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:06.258313 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.258279 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:06.258313 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.258305 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:06.258397 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.258318 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc38c4e-77a6-4175-9033-73067cc44525-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:06.258397 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.258332 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:06.258397 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.258346 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:06.260079 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.260055 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc38c4e-77a6-4175-9033-73067cc44525-istio-token" pod "edc38c4e-77a6-4175-9033-73067cc44525" (UID: "edc38c4e-77a6-4175-9033-73067cc44525"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:06.260719 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.260695 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edc38c4e-77a6-4175-9033-73067cc44525-istio-podinfo" pod "edc38c4e-77a6-4175-9033-73067cc44525" (UID: "edc38c4e-77a6-4175-9033-73067cc44525"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:09:06.260719 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.260702 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-istio-envoy" pod "edc38c4e-77a6-4175-9033-73067cc44525" (UID: "edc38c4e-77a6-4175-9033-73067cc44525"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:06.260868 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.260733 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc38c4e-77a6-4175-9033-73067cc44525-kube-api-access-qc4l5" pod "edc38c4e-77a6-4175-9033-73067cc44525" (UID: "edc38c4e-77a6-4175-9033-73067cc44525"). InnerVolumeSpecName "kube-api-access-qc4l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:06.359222 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.359192 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc38c4e-77a6-4175-9033-73067cc44525-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:06.359222 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.359221 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc38c4e-77a6-4175-9033-73067cc44525-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:06.359515 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.359233 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc4l5\" (UniqueName: \"kubernetes.io/projected/edc38c4e-77a6-4175-9033-73067cc44525-kube-api-access-qc4l5\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:06.359515 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.359247 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc38c4e-77a6-4175-9033-73067cc44525-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:06.360818 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.360780 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww" podUID="d16e3ebb-dc72-474e-b24c-8f92b7335140" containerName="payload-pre-processing" containerID="cri-o://b5bda7d6d39714fe2eab2a859eff84019fd2a7197576ad787027fc800bf8cad2" gracePeriod=25 Nov 01 07:09:06.438139 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.437535 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:06.568317 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.568239 4385 generic.go:356] "Generic (PLEG): container finished" podID="edc38c4e-77a6-4175-9033-73067cc44525" containerID="155fc221a054f53da69cdb9d9ae8056c58ad4c78bb1396a7df063c133c17a362" exitCode=0 Nov 01 07:09:06.568317 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.568281 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" Nov 01 07:09:06.568317 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.568290 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" event={"ID":"edc38c4e-77a6-4175-9033-73067cc44525","Type":"ContainerDied","Data":"155fc221a054f53da69cdb9d9ae8056c58ad4c78bb1396a7df063c133c17a362"} Nov 01 07:09:06.568827 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.568337 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz" event={"ID":"edc38c4e-77a6-4175-9033-73067cc44525","Type":"ContainerDied","Data":"4874e056663ef03274aa736a3542e8a4dd04caaaada26411b51d205559cce372"} Nov 01 07:09:06.568827 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.568365 4385 scope.go:122] "RemoveContainer" containerID="155fc221a054f53da69cdb9d9ae8056c58ad4c78bb1396a7df063c133c17a362" Nov 01 07:09:06.578771 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.578754 4385 scope.go:122] "RemoveContainer" containerID="155fc221a054f53da69cdb9d9ae8056c58ad4c78bb1396a7df063c133c17a362" Nov 01 07:09:06.579096 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:09:06.579077 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"155fc221a054f53da69cdb9d9ae8056c58ad4c78bb1396a7df063c133c17a362\": container with ID starting with 155fc221a054f53da69cdb9d9ae8056c58ad4c78bb1396a7df063c133c17a362 not found: ID does not exist" containerID="155fc221a054f53da69cdb9d9ae8056c58ad4c78bb1396a7df063c133c17a362" Nov 01 07:09:06.579193 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.579103 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"155fc221a054f53da69cdb9d9ae8056c58ad4c78bb1396a7df063c133c17a362"} err="failed to get container status \"155fc221a054f53da69cdb9d9ae8056c58ad4c78bb1396a7df063c133c17a362\": rpc error: code = NotFound desc = could not find container \"155fc221a054f53da69cdb9d9ae8056c58ad4c78bb1396a7df063c133c17a362\": container with ID starting with 155fc221a054f53da69cdb9d9ae8056c58ad4c78bb1396a7df063c133c17a362 not found: ID does not exist" Nov 01 07:09:06.593876 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.593849 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:06.732407 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.732372 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz"] Nov 01 07:09:06.736437 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:06.736414 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-zq6dz"] Nov 01 07:09:06.781016 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:09:06.780977 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:09:07.838136 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:07.838094 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:08.357334 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:08.357302 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:08.383809 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:08.383783 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edc38c4e-77a6-4175-9033-73067cc44525" path="/var/lib/kubelet/pods/edc38c4e-77a6-4175-9033-73067cc44525/volumes" Nov 01 07:09:09.475275 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.475234 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" podUID="27e2ec0f-708a-4ffb-9694-d1030859fc61" containerName="main" containerID="cri-o://5f72ad62f09e45fcdda20c9346d59ffb52ee2ec8d34be89cd1347588d075a679" gracePeriod=45 Nov 01 07:09:09.523746 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.523714 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" podUID="04ae8d7a-c300-4ddc-89a2-7d3400685bda" containerName="main" containerID="cri-o://b51383f646a9128fecd1c27e0f164ceeb92e376a112ca241bc8151b7cb053854" gracePeriod=45 Nov 01 07:09:09.666006 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.665973 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" podUID="27e2ec0f-708a-4ffb-9694-d1030859fc61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": dial tcp 10.134.0.57:8000: connect: connection refused" Nov 01 07:09:09.795514 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.795481 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 07:09:09.822465 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.822440 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 07:09:09.895264 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.895231 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-tmp-dir\") pod \"27e2ec0f-708a-4ffb-9694-d1030859fc61\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " Nov 01 07:09:09.895427 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.895302 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27e2ec0f-708a-4ffb-9694-d1030859fc61-tls-certs\" (UniqueName: \"kubernetes.io/secret/27e2ec0f-708a-4ffb-9694-d1030859fc61-tls-certs\") pod \"27e2ec0f-708a-4ffb-9694-d1030859fc61\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " Nov 01 07:09:09.895427 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.895351 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-model-cache\") pod \"27e2ec0f-708a-4ffb-9694-d1030859fc61\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " Nov 01 07:09:09.895427 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.895410 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04ae8d7a-c300-4ddc-89a2-7d3400685bda-tls-certs\" (UniqueName: \"kubernetes.io/secret/04ae8d7a-c300-4ddc-89a2-7d3400685bda-tls-certs\") pod \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " Nov 01 07:09:09.895601 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.895481 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-home\" (UniqueName: \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-home\") pod \"27e2ec0f-708a-4ffb-9694-d1030859fc61\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " Nov 01 07:09:09.895601 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.895515 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27e2ec0f-708a-4ffb-9694-d1030859fc61-kube-api-access-k8kdh\" (UniqueName: \"kubernetes.io/projected/27e2ec0f-708a-4ffb-9694-d1030859fc61-kube-api-access-k8kdh\") pod \"27e2ec0f-708a-4ffb-9694-d1030859fc61\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " Nov 01 07:09:09.895601 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.895533 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-tmp-dir" pod "27e2ec0f-708a-4ffb-9694-d1030859fc61" (UID: "27e2ec0f-708a-4ffb-9694-d1030859fc61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:09.895601 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.895582 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-dshm\" (UniqueName: \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-dshm\") pod \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " Nov 01 07:09:09.895869 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.895605 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-model-cache" pod "27e2ec0f-708a-4ffb-9694-d1030859fc61" (UID: "27e2ec0f-708a-4ffb-9694-d1030859fc61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:09.895869 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.895617 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-model-cache\") pod \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " Nov 01 07:09:09.895869 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.895651 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-home\" (UniqueName: \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-home\") pod \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " Nov 01 07:09:09.895869 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.895686 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04ae8d7a-c300-4ddc-89a2-7d3400685bda-kube-api-access-6svl5\" (UniqueName: \"kubernetes.io/projected/04ae8d7a-c300-4ddc-89a2-7d3400685bda-kube-api-access-6svl5\") pod \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " Nov 01 07:09:09.895869 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.895720 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-tmp-dir\") pod \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\" (UID: \"04ae8d7a-c300-4ddc-89a2-7d3400685bda\") " Nov 01 07:09:09.895869 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.895763 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-dshm\" (UniqueName: \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-dshm\") pod \"27e2ec0f-708a-4ffb-9694-d1030859fc61\" (UID: \"27e2ec0f-708a-4ffb-9694-d1030859fc61\") " Nov 01 07:09:09.895869 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.895769 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-home" pod "27e2ec0f-708a-4ffb-9694-d1030859fc61" (UID: "27e2ec0f-708a-4ffb-9694-d1030859fc61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:09.896338 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.895927 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-model-cache" pod "04ae8d7a-c300-4ddc-89a2-7d3400685bda" (UID: "04ae8d7a-c300-4ddc-89a2-7d3400685bda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:09.896338 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.896154 4385 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-home\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:09.896338 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.896176 4385 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-model-cache\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:09.896338 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.896190 4385 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-tmp-dir\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:09.896338 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.896206 4385 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-model-cache\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:09.896614 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.896569 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-home" pod "04ae8d7a-c300-4ddc-89a2-7d3400685bda" (UID: "04ae8d7a-c300-4ddc-89a2-7d3400685bda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:09.896857 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.896635 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-tmp-dir" pod "04ae8d7a-c300-4ddc-89a2-7d3400685bda" (UID: "04ae8d7a-c300-4ddc-89a2-7d3400685bda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:09.898714 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.898679 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-dshm" pod "27e2ec0f-708a-4ffb-9694-d1030859fc61" (UID: "27e2ec0f-708a-4ffb-9694-d1030859fc61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:09.898975 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.898951 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27e2ec0f-708a-4ffb-9694-d1030859fc61-tls-certs" pod "27e2ec0f-708a-4ffb-9694-d1030859fc61" (UID: "27e2ec0f-708a-4ffb-9694-d1030859fc61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:09:09.898975 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.898961 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04ae8d7a-c300-4ddc-89a2-7d3400685bda-tls-certs" pod "04ae8d7a-c300-4ddc-89a2-7d3400685bda" (UID: "04ae8d7a-c300-4ddc-89a2-7d3400685bda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:09:09.899149 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.898989 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ae8d7a-c300-4ddc-89a2-7d3400685bda-kube-api-access-6svl5" pod "04ae8d7a-c300-4ddc-89a2-7d3400685bda" (UID: "04ae8d7a-c300-4ddc-89a2-7d3400685bda"). InnerVolumeSpecName "kube-api-access-6svl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:09.899149 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.899010 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e2ec0f-708a-4ffb-9694-d1030859fc61-kube-api-access-k8kdh" pod "27e2ec0f-708a-4ffb-9694-d1030859fc61" (UID: "27e2ec0f-708a-4ffb-9694-d1030859fc61"). InnerVolumeSpecName "kube-api-access-k8kdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:09.899231 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.899200 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-dshm" pod "04ae8d7a-c300-4ddc-89a2-7d3400685bda" (UID: "04ae8d7a-c300-4ddc-89a2-7d3400685bda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:09.997777 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.997746 4385 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-dshm\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:09.997777 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.997770 4385 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-home\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:09.997777 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.997780 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6svl5\" (UniqueName: \"kubernetes.io/projected/04ae8d7a-c300-4ddc-89a2-7d3400685bda-kube-api-access-6svl5\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:09.998032 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.997790 4385 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04ae8d7a-c300-4ddc-89a2-7d3400685bda-tmp-dir\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:09.998032 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.997800 4385 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27e2ec0f-708a-4ffb-9694-d1030859fc61-dshm\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:09.998032 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.997808 4385 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27e2ec0f-708a-4ffb-9694-d1030859fc61-tls-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:09.998032 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.997815 4385 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04ae8d7a-c300-4ddc-89a2-7d3400685bda-tls-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:09.998032 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:09.997823 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8kdh\" (UniqueName: \"kubernetes.io/projected/27e2ec0f-708a-4ffb-9694-d1030859fc61-kube-api-access-k8kdh\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:10.594175 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:10.594132 4385 generic.go:356] "Generic (PLEG): container finished" podID="27e2ec0f-708a-4ffb-9694-d1030859fc61" containerID="5f72ad62f09e45fcdda20c9346d59ffb52ee2ec8d34be89cd1347588d075a679" exitCode=0 Nov 01 07:09:10.594661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:10.594194 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" Nov 01 07:09:10.594661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:10.594226 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" event={"ID":"27e2ec0f-708a-4ffb-9694-d1030859fc61","Type":"ContainerDied","Data":"5f72ad62f09e45fcdda20c9346d59ffb52ee2ec8d34be89cd1347588d075a679"} Nov 01 07:09:10.594661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:10.594274 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2" event={"ID":"27e2ec0f-708a-4ffb-9694-d1030859fc61","Type":"ContainerDied","Data":"282e5fa97115d2bc47d00be007a7c0bce6a90a90028733c30221161fe228c997"} Nov 01 07:09:10.594661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:10.594297 4385 scope.go:122] "RemoveContainer" containerID="5f72ad62f09e45fcdda20c9346d59ffb52ee2ec8d34be89cd1347588d075a679" Nov 01 07:09:10.598876 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:10.598280 4385 generic.go:356] "Generic (PLEG): container finished" podID="04ae8d7a-c300-4ddc-89a2-7d3400685bda" containerID="b51383f646a9128fecd1c27e0f164ceeb92e376a112ca241bc8151b7cb053854" exitCode=0 Nov 01 07:09:10.598876 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:10.598328 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" event={"ID":"04ae8d7a-c300-4ddc-89a2-7d3400685bda","Type":"ContainerDied","Data":"b51383f646a9128fecd1c27e0f164ceeb92e376a112ca241bc8151b7cb053854"} Nov 01 07:09:10.598876 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:10.598378 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" event={"ID":"04ae8d7a-c300-4ddc-89a2-7d3400685bda","Type":"ContainerDied","Data":"12d95a6042ac72f5538caf34d21a513c6784a6d834f256df9ff011d7ef1f4ee6"} Nov 01 07:09:10.598876 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:10.598405 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd" Nov 01 07:09:10.622942 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:10.622909 4385 scope.go:122] "RemoveContainer" containerID="5f72ad62f09e45fcdda20c9346d59ffb52ee2ec8d34be89cd1347588d075a679" Nov 01 07:09:10.623321 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:09:10.623281 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f72ad62f09e45fcdda20c9346d59ffb52ee2ec8d34be89cd1347588d075a679\": container with ID starting with 5f72ad62f09e45fcdda20c9346d59ffb52ee2ec8d34be89cd1347588d075a679 not found: ID does not exist" containerID="5f72ad62f09e45fcdda20c9346d59ffb52ee2ec8d34be89cd1347588d075a679" Nov 01 07:09:10.623401 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:10.623331 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f72ad62f09e45fcdda20c9346d59ffb52ee2ec8d34be89cd1347588d075a679"} err="failed to get container status \"5f72ad62f09e45fcdda20c9346d59ffb52ee2ec8d34be89cd1347588d075a679\": rpc error: code = NotFound desc = could not find container \"5f72ad62f09e45fcdda20c9346d59ffb52ee2ec8d34be89cd1347588d075a679\": container with ID starting with 5f72ad62f09e45fcdda20c9346d59ffb52ee2ec8d34be89cd1347588d075a679 not found: ID does not exist" Nov 01 07:09:10.623401 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:10.623348 4385 scope.go:122] "RemoveContainer" containerID="b51383f646a9128fecd1c27e0f164ceeb92e376a112ca241bc8151b7cb053854" Nov 01 07:09:10.633065 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:10.633042 4385 scope.go:122] "RemoveContainer" containerID="b51383f646a9128fecd1c27e0f164ceeb92e376a112ca241bc8151b7cb053854" Nov 01 07:09:10.633365 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:09:10.633341 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b51383f646a9128fecd1c27e0f164ceeb92e376a112ca241bc8151b7cb053854\": container with ID starting with b51383f646a9128fecd1c27e0f164ceeb92e376a112ca241bc8151b7cb053854 not found: ID does not exist" containerID="b51383f646a9128fecd1c27e0f164ceeb92e376a112ca241bc8151b7cb053854" Nov 01 07:09:10.633423 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:10.633375 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b51383f646a9128fecd1c27e0f164ceeb92e376a112ca241bc8151b7cb053854"} err="failed to get container status \"b51383f646a9128fecd1c27e0f164ceeb92e376a112ca241bc8151b7cb053854\": rpc error: code = NotFound desc = could not find container \"b51383f646a9128fecd1c27e0f164ceeb92e376a112ca241bc8151b7cb053854\": container with ID starting with b51383f646a9128fecd1c27e0f164ceeb92e376a112ca241bc8151b7cb053854 not found: ID does not exist" Nov 01 07:09:10.760617 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:10.760536 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2"] Nov 01 07:09:10.812085 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:10.812052 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/premium-w0-3ba2f3-kserve-5cd7896897-bxdj2"] Nov 01 07:09:10.917156 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:10.917120 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd"] Nov 01 07:09:10.921119 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:10.921091 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3ba2f3/simulator-w0-3ba2f3-kserve-6cb5bf75bd-6bdmd"] Nov 01 07:09:10.954442 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:10.954409 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:10.980976 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:10.980942 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" podUID="d0029faa-cd51-43d8-8c9f-02399257ea72" containerName="main" containerID="cri-o://566d6efd0d2778af3c0370d57b15d242426064a7d8cb8687684e103785500c4b" gracePeriod=45 Nov 01 07:09:11.032109 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.032039 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" podUID="f5f0b488-c1f8-4a7e-9779-ebf4d94ad037" containerName="main" containerID="cri-o://0cd7d8c77186041bfa48a8177e7eed4e6d72e9f94da09402e031ed57d8d2b353" gracePeriod=45 Nov 01 07:09:11.038107 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.038075 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" podUID="6571db97-e767-4917-9543-5ebd4e6ff1e5" containerName="main" containerID="cri-o://2f0d8191dc1ce96678b6fd8843f9d9cff6769fe4e18d72512402eed1f099027c" gracePeriod=45 Nov 01 07:09:11.307479 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.307406 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 07:09:11.331486 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.331457 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 07:09:11.340972 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.340951 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 07:09:11.412717 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.412682 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-home\" (UniqueName: \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-home\") pod \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " Nov 01 07:09:11.412949 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.412724 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-dshm\") pod \"6571db97-e767-4917-9543-5ebd4e6ff1e5\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " Nov 01 07:09:11.412949 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.412756 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-model-cache\") pod \"d0029faa-cd51-43d8-8c9f-02399257ea72\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " Nov 01 07:09:11.412949 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.412792 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-home\" (UniqueName: \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-home\") pod \"6571db97-e767-4917-9543-5ebd4e6ff1e5\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " Nov 01 07:09:11.412949 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.412878 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-tmp-dir\") pod \"6571db97-e767-4917-9543-5ebd4e6ff1e5\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " Nov 01 07:09:11.412949 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.412909 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-tmp-dir\") pod \"d0029faa-cd51-43d8-8c9f-02399257ea72\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " Nov 01 07:09:11.412949 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.412942 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-dshm\") pod \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " Nov 01 07:09:11.413291 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.413009 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-home\" (UniqueName: \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-home\") pod \"d0029faa-cd51-43d8-8c9f-02399257ea72\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " Nov 01 07:09:11.413291 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.413017 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-home" pod "f5f0b488-c1f8-4a7e-9779-ebf4d94ad037" (UID: "f5f0b488-c1f8-4a7e-9779-ebf4d94ad037"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:11.413291 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.413059 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0029faa-cd51-43d8-8c9f-02399257ea72-kube-api-access-mqljn\" (UniqueName: \"kubernetes.io/projected/d0029faa-cd51-43d8-8c9f-02399257ea72-kube-api-access-mqljn\") pod \"d0029faa-cd51-43d8-8c9f-02399257ea72\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " Nov 01 07:09:11.413291 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.413100 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-tmp-dir\") pod \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " Nov 01 07:09:11.413291 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.413142 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-model-cache\") pod \"6571db97-e767-4917-9543-5ebd4e6ff1e5\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " Nov 01 07:09:11.413291 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.413208 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-tls-certs\") pod \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " Nov 01 07:09:11.413291 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.413238 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-dshm\") pod \"d0029faa-cd51-43d8-8c9f-02399257ea72\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " Nov 01 07:09:11.413291 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.413238 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-tmp-dir" pod "d0029faa-cd51-43d8-8c9f-02399257ea72" (UID: "d0029faa-cd51-43d8-8c9f-02399257ea72"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:11.413291 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.413283 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-model-cache\") pod \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " Nov 01 07:09:11.413773 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.413330 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6571db97-e767-4917-9543-5ebd4e6ff1e5-kube-api-access-4mvpj\" (UniqueName: \"kubernetes.io/projected/6571db97-e767-4917-9543-5ebd4e6ff1e5-kube-api-access-4mvpj\") pod \"6571db97-e767-4917-9543-5ebd4e6ff1e5\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " Nov 01 07:09:11.413773 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.413372 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6571db97-e767-4917-9543-5ebd4e6ff1e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/6571db97-e767-4917-9543-5ebd4e6ff1e5-tls-certs\") pod \"6571db97-e767-4917-9543-5ebd4e6ff1e5\" (UID: \"6571db97-e767-4917-9543-5ebd4e6ff1e5\") " Nov 01 07:09:11.413773 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.413407 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-home" pod "6571db97-e767-4917-9543-5ebd4e6ff1e5" (UID: "6571db97-e767-4917-9543-5ebd4e6ff1e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:11.413773 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.413410 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0029faa-cd51-43d8-8c9f-02399257ea72-tls-certs\" (UniqueName: \"kubernetes.io/secret/d0029faa-cd51-43d8-8c9f-02399257ea72-tls-certs\") pod \"d0029faa-cd51-43d8-8c9f-02399257ea72\" (UID: \"d0029faa-cd51-43d8-8c9f-02399257ea72\") " Nov 01 07:09:11.413773 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.413499 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-kube-api-access-shxkf\" (UniqueName: \"kubernetes.io/projected/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-kube-api-access-shxkf\") pod \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\" (UID: \"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037\") " Nov 01 07:09:11.413773 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.413565 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-model-cache" pod "6571db97-e767-4917-9543-5ebd4e6ff1e5" (UID: "6571db97-e767-4917-9543-5ebd4e6ff1e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:11.413773 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.413667 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-home" pod "d0029faa-cd51-43d8-8c9f-02399257ea72" (UID: "d0029faa-cd51-43d8-8c9f-02399257ea72"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:11.413773 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.413685 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-model-cache" pod "f5f0b488-c1f8-4a7e-9779-ebf4d94ad037" (UID: "f5f0b488-c1f8-4a7e-9779-ebf4d94ad037"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:11.414443 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.413921 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-tmp-dir" pod "6571db97-e767-4917-9543-5ebd4e6ff1e5" (UID: "6571db97-e767-4917-9543-5ebd4e6ff1e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:11.414443 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.413979 4385 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-tmp-dir\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:11.414443 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.414001 4385 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-home\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:11.414443 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.414014 4385 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-model-cache\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:11.414443 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.414040 4385 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-model-cache\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:11.414443 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.414054 4385 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-home\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:11.414443 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.414066 4385 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-home\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:11.414443 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.414116 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-model-cache" pod "d0029faa-cd51-43d8-8c9f-02399257ea72" (UID: "d0029faa-cd51-43d8-8c9f-02399257ea72"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:11.414945 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.414923 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-tmp-dir" pod "f5f0b488-c1f8-4a7e-9779-ebf4d94ad037" (UID: "f5f0b488-c1f8-4a7e-9779-ebf4d94ad037"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:11.416784 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.416744 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0029faa-cd51-43d8-8c9f-02399257ea72-tls-certs" pod "d0029faa-cd51-43d8-8c9f-02399257ea72" (UID: "d0029faa-cd51-43d8-8c9f-02399257ea72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:09:11.417117 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.417094 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-dshm" pod "f5f0b488-c1f8-4a7e-9779-ebf4d94ad037" (UID: "f5f0b488-c1f8-4a7e-9779-ebf4d94ad037"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:11.418281 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.418258 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-tls-certs" pod "f5f0b488-c1f8-4a7e-9779-ebf4d94ad037" (UID: "f5f0b488-c1f8-4a7e-9779-ebf4d94ad037"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:09:11.418281 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.418272 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-kube-api-access-shxkf" pod "f5f0b488-c1f8-4a7e-9779-ebf4d94ad037" (UID: "f5f0b488-c1f8-4a7e-9779-ebf4d94ad037"). InnerVolumeSpecName "kube-api-access-shxkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:11.418413 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.418317 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0029faa-cd51-43d8-8c9f-02399257ea72-kube-api-access-mqljn" pod "d0029faa-cd51-43d8-8c9f-02399257ea72" (UID: "d0029faa-cd51-43d8-8c9f-02399257ea72"). InnerVolumeSpecName "kube-api-access-mqljn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:11.418679 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.418644 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-dshm" pod "6571db97-e767-4917-9543-5ebd4e6ff1e5" (UID: "6571db97-e767-4917-9543-5ebd4e6ff1e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:11.418755 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.418727 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-dshm" pod "d0029faa-cd51-43d8-8c9f-02399257ea72" (UID: "d0029faa-cd51-43d8-8c9f-02399257ea72"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:11.418803 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.418749 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6571db97-e767-4917-9543-5ebd4e6ff1e5-tls-certs" pod "6571db97-e767-4917-9543-5ebd4e6ff1e5" (UID: "6571db97-e767-4917-9543-5ebd4e6ff1e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:09:11.419116 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.419099 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6571db97-e767-4917-9543-5ebd4e6ff1e5-kube-api-access-4mvpj" pod "6571db97-e767-4917-9543-5ebd4e6ff1e5" (UID: "6571db97-e767-4917-9543-5ebd4e6ff1e5"). InnerVolumeSpecName "kube-api-access-4mvpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:11.514995 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.514966 4385 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-tls-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:11.514995 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.514992 4385 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-dshm\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:11.514995 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.515002 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mvpj\" (UniqueName: \"kubernetes.io/projected/6571db97-e767-4917-9543-5ebd4e6ff1e5-kube-api-access-4mvpj\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:11.515228 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.515013 4385 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6571db97-e767-4917-9543-5ebd4e6ff1e5-tls-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:11.515228 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.515021 4385 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0029faa-cd51-43d8-8c9f-02399257ea72-tls-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:11.515228 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.515029 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shxkf\" (UniqueName: \"kubernetes.io/projected/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-kube-api-access-shxkf\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:11.515228 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.515037 4385 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-dshm\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:11.515228 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.515045 4385 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0029faa-cd51-43d8-8c9f-02399257ea72-model-cache\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:11.515228 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.515053 4385 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6571db97-e767-4917-9543-5ebd4e6ff1e5-tmp-dir\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:11.515228 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.515062 4385 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-dshm\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:11.515228 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.515070 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqljn\" (UniqueName: \"kubernetes.io/projected/d0029faa-cd51-43d8-8c9f-02399257ea72-kube-api-access-mqljn\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:11.515228 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.515078 4385 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037-tmp-dir\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:11.608359 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.608329 4385 generic.go:356] "Generic (PLEG): container finished" podID="f5f0b488-c1f8-4a7e-9779-ebf4d94ad037" containerID="0cd7d8c77186041bfa48a8177e7eed4e6d72e9f94da09402e031ed57d8d2b353" exitCode=0 Nov 01 07:09:11.608874 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.608370 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" Nov 01 07:09:11.608874 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.608413 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" event={"ID":"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037","Type":"ContainerDied","Data":"0cd7d8c77186041bfa48a8177e7eed4e6d72e9f94da09402e031ed57d8d2b353"} Nov 01 07:09:11.608874 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.608458 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl" event={"ID":"f5f0b488-c1f8-4a7e-9779-ebf4d94ad037","Type":"ContainerDied","Data":"3cc6b350fef32e8810d25f7ab9834762c8c5d4edb6b6588238a068e20d38cf67"} Nov 01 07:09:11.608874 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.608479 4385 scope.go:122] "RemoveContainer" containerID="0cd7d8c77186041bfa48a8177e7eed4e6d72e9f94da09402e031ed57d8d2b353" Nov 01 07:09:11.610642 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.610375 4385 generic.go:356] "Generic (PLEG): container finished" podID="d0029faa-cd51-43d8-8c9f-02399257ea72" containerID="566d6efd0d2778af3c0370d57b15d242426064a7d8cb8687684e103785500c4b" exitCode=0 Nov 01 07:09:11.610642 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.610419 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" Nov 01 07:09:11.610642 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.610451 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" event={"ID":"d0029faa-cd51-43d8-8c9f-02399257ea72","Type":"ContainerDied","Data":"566d6efd0d2778af3c0370d57b15d242426064a7d8cb8687684e103785500c4b"} Nov 01 07:09:11.610642 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.610495 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7" event={"ID":"d0029faa-cd51-43d8-8c9f-02399257ea72","Type":"ContainerDied","Data":"7439f0b030aaa7e719a15cec21ffe782d6e4eaa31eb423086efa46e8add43d4d"} Nov 01 07:09:11.613356 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.613332 4385 generic.go:356] "Generic (PLEG): container finished" podID="6571db97-e767-4917-9543-5ebd4e6ff1e5" containerID="2f0d8191dc1ce96678b6fd8843f9d9cff6769fe4e18d72512402eed1f099027c" exitCode=0 Nov 01 07:09:11.613475 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.613375 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" Nov 01 07:09:11.613475 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.613422 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" event={"ID":"6571db97-e767-4917-9543-5ebd4e6ff1e5","Type":"ContainerDied","Data":"2f0d8191dc1ce96678b6fd8843f9d9cff6769fe4e18d72512402eed1f099027c"} Nov 01 07:09:11.613475 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.613464 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx" event={"ID":"6571db97-e767-4917-9543-5ebd4e6ff1e5","Type":"ContainerDied","Data":"46e283e7163d409e3ded396121a9e308b3bf38eb8331c16d6219ca885273385b"} Nov 01 07:09:11.620780 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.620768 4385 scope.go:122] "RemoveContainer" containerID="0cd7d8c77186041bfa48a8177e7eed4e6d72e9f94da09402e031ed57d8d2b353" Nov 01 07:09:11.621102 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:09:11.621081 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cd7d8c77186041bfa48a8177e7eed4e6d72e9f94da09402e031ed57d8d2b353\": container with ID starting with 0cd7d8c77186041bfa48a8177e7eed4e6d72e9f94da09402e031ed57d8d2b353 not found: ID does not exist" containerID="0cd7d8c77186041bfa48a8177e7eed4e6d72e9f94da09402e031ed57d8d2b353" Nov 01 07:09:11.621197 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.621117 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd7d8c77186041bfa48a8177e7eed4e6d72e9f94da09402e031ed57d8d2b353"} err="failed to get container status \"0cd7d8c77186041bfa48a8177e7eed4e6d72e9f94da09402e031ed57d8d2b353\": rpc error: code = NotFound desc = could not find container \"0cd7d8c77186041bfa48a8177e7eed4e6d72e9f94da09402e031ed57d8d2b353\": container with ID starting with 0cd7d8c77186041bfa48a8177e7eed4e6d72e9f94da09402e031ed57d8d2b353 not found: ID does not exist" Nov 01 07:09:11.621197 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.621138 4385 scope.go:122] "RemoveContainer" containerID="566d6efd0d2778af3c0370d57b15d242426064a7d8cb8687684e103785500c4b" Nov 01 07:09:11.630030 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.630005 4385 scope.go:122] "RemoveContainer" containerID="566d6efd0d2778af3c0370d57b15d242426064a7d8cb8687684e103785500c4b" Nov 01 07:09:11.630284 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:09:11.630268 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"566d6efd0d2778af3c0370d57b15d242426064a7d8cb8687684e103785500c4b\": container with ID starting with 566d6efd0d2778af3c0370d57b15d242426064a7d8cb8687684e103785500c4b not found: ID does not exist" containerID="566d6efd0d2778af3c0370d57b15d242426064a7d8cb8687684e103785500c4b" Nov 01 07:09:11.630328 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.630292 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566d6efd0d2778af3c0370d57b15d242426064a7d8cb8687684e103785500c4b"} err="failed to get container status \"566d6efd0d2778af3c0370d57b15d242426064a7d8cb8687684e103785500c4b\": rpc error: code = NotFound desc = could not find container \"566d6efd0d2778af3c0370d57b15d242426064a7d8cb8687684e103785500c4b\": container with ID starting with 566d6efd0d2778af3c0370d57b15d242426064a7d8cb8687684e103785500c4b not found: ID does not exist" Nov 01 07:09:11.630328 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.630309 4385 scope.go:122] "RemoveContainer" containerID="2f0d8191dc1ce96678b6fd8843f9d9cff6769fe4e18d72512402eed1f099027c" Nov 01 07:09:11.638986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.638968 4385 scope.go:122] "RemoveContainer" containerID="2f0d8191dc1ce96678b6fd8843f9d9cff6769fe4e18d72512402eed1f099027c" Nov 01 07:09:11.639240 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:09:11.639221 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0d8191dc1ce96678b6fd8843f9d9cff6769fe4e18d72512402eed1f099027c\": container with ID starting with 2f0d8191dc1ce96678b6fd8843f9d9cff6769fe4e18d72512402eed1f099027c not found: ID does not exist" containerID="2f0d8191dc1ce96678b6fd8843f9d9cff6769fe4e18d72512402eed1f099027c" Nov 01 07:09:11.639297 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.639248 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0d8191dc1ce96678b6fd8843f9d9cff6769fe4e18d72512402eed1f099027c"} err="failed to get container status \"2f0d8191dc1ce96678b6fd8843f9d9cff6769fe4e18d72512402eed1f099027c\": rpc error: code = NotFound desc = could not find container \"2f0d8191dc1ce96678b6fd8843f9d9cff6769fe4e18d72512402eed1f099027c\": container with ID starting with 2f0d8191dc1ce96678b6fd8843f9d9cff6769fe4e18d72512402eed1f099027c not found: ID does not exist" Nov 01 07:09:11.760436 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.760408 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl"] Nov 01 07:09:11.763172 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.763152 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-190724/premium-w4-190724-kserve-8599fcf989-dlzpl"] Nov 01 07:09:11.869319 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.869240 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7"] Nov 01 07:09:11.871853 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.871817 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-190724/distinct-w4-190724-kserve-5f958f968d-zsnz7"] Nov 01 07:09:11.977735 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.977703 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx"] Nov 01 07:09:11.980434 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:11.980409 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-190724/simulator-w4-190724-kserve-5f758995c6-nz6xx"] Nov 01 07:09:12.384753 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:12.384722 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04ae8d7a-c300-4ddc-89a2-7d3400685bda" path="/var/lib/kubelet/pods/04ae8d7a-c300-4ddc-89a2-7d3400685bda/volumes" Nov 01 07:09:12.385172 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:12.385158 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27e2ec0f-708a-4ffb-9694-d1030859fc61" path="/var/lib/kubelet/pods/27e2ec0f-708a-4ffb-9694-d1030859fc61/volumes" Nov 01 07:09:12.385527 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:12.385515 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6571db97-e767-4917-9543-5ebd4e6ff1e5" path="/var/lib/kubelet/pods/6571db97-e767-4917-9543-5ebd4e6ff1e5/volumes" Nov 01 07:09:12.386214 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:12.386199 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0029faa-cd51-43d8-8c9f-02399257ea72" path="/var/lib/kubelet/pods/d0029faa-cd51-43d8-8c9f-02399257ea72/volumes" Nov 01 07:09:12.386618 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:12.386605 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5f0b488-c1f8-4a7e-9779-ebf4d94ad037" path="/var/lib/kubelet/pods/f5f0b488-c1f8-4a7e-9779-ebf4d94ad037/volumes" Nov 01 07:09:12.726113 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:12.726036 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:13.792657 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:13.792622 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:14.209127 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:14.209093 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww" podUID="d16e3ebb-dc72-474e-b24c-8f92b7335140" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:09:16.654106 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:16.654069 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:16.855097 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:16.855057 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:18.531634 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:18.531546 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:22.231400 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:22.231366 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:22.377245 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:22.377212 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:23.238815 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:23.238781 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:24.208746 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:24.208713 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww" podUID="d16e3ebb-dc72-474e-b24c-8f92b7335140" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:09:26.611594 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:26.611557 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww" Nov 01 07:09:26.674818 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:26.674744 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d16e3ebb-dc72-474e-b24c-8f92b7335140-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d16e3ebb-dc72-474e-b24c-8f92b7335140-plugins-config-volume\") pod \"d16e3ebb-dc72-474e-b24c-8f92b7335140\" (UID: \"d16e3ebb-dc72-474e-b24c-8f92b7335140\") " Nov 01 07:09:26.674985 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:26.674845 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16e3ebb-dc72-474e-b24c-8f92b7335140-kube-api-access-jnfxg\" (UniqueName: \"kubernetes.io/projected/d16e3ebb-dc72-474e-b24c-8f92b7335140-kube-api-access-jnfxg\") pod \"d16e3ebb-dc72-474e-b24c-8f92b7335140\" (UID: \"d16e3ebb-dc72-474e-b24c-8f92b7335140\") " Nov 01 07:09:26.675131 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:26.675108 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16e3ebb-dc72-474e-b24c-8f92b7335140-plugins-config-volume" pod "d16e3ebb-dc72-474e-b24c-8f92b7335140" (UID: "d16e3ebb-dc72-474e-b24c-8f92b7335140"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:09:26.677133 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:26.677109 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16e3ebb-dc72-474e-b24c-8f92b7335140-kube-api-access-jnfxg" pod "d16e3ebb-dc72-474e-b24c-8f92b7335140" (UID: "d16e3ebb-dc72-474e-b24c-8f92b7335140"). InnerVolumeSpecName "kube-api-access-jnfxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:26.712797 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:26.712735 4385 generic.go:356] "Generic (PLEG): container finished" podID="d16e3ebb-dc72-474e-b24c-8f92b7335140" containerID="b5bda7d6d39714fe2eab2a859eff84019fd2a7197576ad787027fc800bf8cad2" exitCode=0 Nov 01 07:09:26.712962 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:26.712797 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww" Nov 01 07:09:26.712962 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:26.712821 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww" event={"ID":"d16e3ebb-dc72-474e-b24c-8f92b7335140","Type":"ContainerDied","Data":"b5bda7d6d39714fe2eab2a859eff84019fd2a7197576ad787027fc800bf8cad2"} Nov 01 07:09:26.712962 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:26.712900 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww" event={"ID":"d16e3ebb-dc72-474e-b24c-8f92b7335140","Type":"ContainerDied","Data":"3a84783b07ede293e364dcb0b729e3b6af5af469ef1089ab6748ba98da518859"} Nov 01 07:09:26.712962 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:26.712932 4385 scope.go:122] "RemoveContainer" containerID="b5bda7d6d39714fe2eab2a859eff84019fd2a7197576ad787027fc800bf8cad2" Nov 01 07:09:26.728404 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:26.728330 4385 scope.go:122] "RemoveContainer" containerID="b5bda7d6d39714fe2eab2a859eff84019fd2a7197576ad787027fc800bf8cad2" Nov 01 07:09:26.729152 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:09:26.729125 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5bda7d6d39714fe2eab2a859eff84019fd2a7197576ad787027fc800bf8cad2\": container with ID starting with b5bda7d6d39714fe2eab2a859eff84019fd2a7197576ad787027fc800bf8cad2 not found: ID does not exist" containerID="b5bda7d6d39714fe2eab2a859eff84019fd2a7197576ad787027fc800bf8cad2" Nov 01 07:09:26.729280 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:26.729164 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5bda7d6d39714fe2eab2a859eff84019fd2a7197576ad787027fc800bf8cad2"} err="failed to get container status \"b5bda7d6d39714fe2eab2a859eff84019fd2a7197576ad787027fc800bf8cad2\": rpc error: code = NotFound desc = could not find container \"b5bda7d6d39714fe2eab2a859eff84019fd2a7197576ad787027fc800bf8cad2\": container with ID starting with b5bda7d6d39714fe2eab2a859eff84019fd2a7197576ad787027fc800bf8cad2 not found: ID does not exist" Nov 01 07:09:26.775824 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:26.775793 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d16e3ebb-dc72-474e-b24c-8f92b7335140-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:26.776002 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:26.775823 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnfxg\" (UniqueName: \"kubernetes.io/projected/d16e3ebb-dc72-474e-b24c-8f92b7335140-kube-api-access-jnfxg\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:26.846034 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:26.845993 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww"] Nov 01 07:09:26.850377 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:26.850342 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3ba2f3-5579b8f879-nkpww"] Nov 01 07:09:28.043659 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:28.043623 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:28.385990 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:28.385954 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d16e3ebb-dc72-474e-b24c-8f92b7335140" path="/var/lib/kubelet/pods/d16e3ebb-dc72-474e-b24c-8f92b7335140/volumes" Nov 01 07:09:29.405607 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.405572 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j"] Nov 01 07:09:29.406054 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.405996 4385 cpu_manager.go:414] "containerMap: removing container" podUID="04ae8d7a-c300-4ddc-89a2-7d3400685bda" containerName="main" Nov 01 07:09:29.406054 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406007 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04ae8d7a-c300-4ddc-89a2-7d3400685bda" containerName="main" Nov 01 07:09:29.406054 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406029 4385 cpu_manager.go:414] "containerMap: removing container" podUID="94a78d83-1b31-4176-95eb-b3b03b076419" containerName="payload-pre-processing" Nov 01 07:09:29.406054 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406034 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a78d83-1b31-4176-95eb-b3b03b076419" containerName="payload-pre-processing" Nov 01 07:09:29.406054 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406043 4385 cpu_manager.go:414] "containerMap: removing container" podUID="f35d3f2c-2ea5-47d6-b808-93a12e4923c0" containerName="main" Nov 01 07:09:29.406054 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406049 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f35d3f2c-2ea5-47d6-b808-93a12e4923c0" containerName="main" Nov 01 07:09:29.406054 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406055 4385 cpu_manager.go:414] "containerMap: removing container" podUID="3c777717-c462-4ba3-bdc3-7b423808eb9b" containerName="istio-proxy" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406060 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c777717-c462-4ba3-bdc3-7b423808eb9b" containerName="istio-proxy" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406072 4385 cpu_manager.go:414] "containerMap: removing container" podUID="f5f0b488-c1f8-4a7e-9779-ebf4d94ad037" containerName="main" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406077 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5f0b488-c1f8-4a7e-9779-ebf4d94ad037" containerName="main" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406087 4385 cpu_manager.go:414] "containerMap: removing container" podUID="695583a3-4ef8-4cad-ba76-69791506f2f3" containerName="main" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406091 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="695583a3-4ef8-4cad-ba76-69791506f2f3" containerName="main" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406097 4385 cpu_manager.go:414] "containerMap: removing container" podUID="3c777717-c462-4ba3-bdc3-7b423808eb9b" containerName="istio-proxy" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406102 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c777717-c462-4ba3-bdc3-7b423808eb9b" containerName="istio-proxy" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406112 4385 cpu_manager.go:414] "containerMap: removing container" podUID="1a10a1d9-d645-42fe-9aa1-d79a35398d68" containerName="payload-processing" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406117 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a10a1d9-d645-42fe-9aa1-d79a35398d68" containerName="payload-processing" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406127 4385 cpu_manager.go:414] "containerMap: removing container" podUID="d0029faa-cd51-43d8-8c9f-02399257ea72" containerName="main" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406131 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0029faa-cd51-43d8-8c9f-02399257ea72" containerName="main" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406140 4385 cpu_manager.go:414] "containerMap: removing container" podUID="27e2ec0f-708a-4ffb-9694-d1030859fc61" containerName="main" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406145 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e2ec0f-708a-4ffb-9694-d1030859fc61" containerName="main" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406152 4385 cpu_manager.go:414] "containerMap: removing container" podUID="4001c2de-286a-4407-9c74-0cc2492bc8ca" containerName="main" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406156 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4001c2de-286a-4407-9c74-0cc2492bc8ca" containerName="main" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406163 4385 cpu_manager.go:414] "containerMap: removing container" podUID="d16e3ebb-dc72-474e-b24c-8f92b7335140" containerName="payload-pre-processing" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406167 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16e3ebb-dc72-474e-b24c-8f92b7335140" containerName="payload-pre-processing" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406176 4385 cpu_manager.go:414] "containerMap: removing container" podUID="6571db97-e767-4917-9543-5ebd4e6ff1e5" containerName="main" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406180 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6571db97-e767-4917-9543-5ebd4e6ff1e5" containerName="main" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406185 4385 cpu_manager.go:414] "containerMap: removing container" podUID="edc38c4e-77a6-4175-9033-73067cc44525" containerName="istio-proxy" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406190 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edc38c4e-77a6-4175-9033-73067cc44525" containerName="istio-proxy" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406247 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0029faa-cd51-43d8-8c9f-02399257ea72" containerName="main" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406256 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="27e2ec0f-708a-4ffb-9694-d1030859fc61" containerName="main" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406263 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="4001c2de-286a-4407-9c74-0cc2492bc8ca" containerName="main" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406270 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16e3ebb-dc72-474e-b24c-8f92b7335140" containerName="payload-pre-processing" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406278 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="6571db97-e767-4917-9543-5ebd4e6ff1e5" containerName="main" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406284 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="edc38c4e-77a6-4175-9033-73067cc44525" containerName="istio-proxy" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406293 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="04ae8d7a-c300-4ddc-89a2-7d3400685bda" containerName="main" Nov 01 07:09:29.406289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406303 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a78d83-1b31-4176-95eb-b3b03b076419" containerName="payload-pre-processing" Nov 01 07:09:29.407193 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406311 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="f35d3f2c-2ea5-47d6-b808-93a12e4923c0" containerName="main" Nov 01 07:09:29.407193 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406319 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c777717-c462-4ba3-bdc3-7b423808eb9b" containerName="istio-proxy" Nov 01 07:09:29.407193 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406329 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5f0b488-c1f8-4a7e-9779-ebf4d94ad037" containerName="main" Nov 01 07:09:29.407193 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406339 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="695583a3-4ef8-4cad-ba76-69791506f2f3" containerName="main" Nov 01 07:09:29.407193 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406346 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c777717-c462-4ba3-bdc3-7b423808eb9b" containerName="istio-proxy" Nov 01 07:09:29.407193 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.406352 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a10a1d9-d645-42fe-9aa1-d79a35398d68" containerName="payload-processing" Nov 01 07:09:29.410809 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.410793 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.455638 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.455605 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt"] Nov 01 07:09:29.459856 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.459811 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.505975 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.505944 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.506167 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.506010 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm2b2\" (UniqueName: \"kubernetes.io/projected/29951118-24dc-49e4-bdc0-11f45296dcba-kube-api-access-jm2b2\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.506167 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.506148 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.506289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.506183 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29951118-24dc-49e4-bdc0-11f45296dcba-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.506289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.506256 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.506469 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.506291 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29951118-24dc-49e4-bdc0-11f45296dcba-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.506469 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.506335 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.506469 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.506416 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.506469 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.506444 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29951118-24dc-49e4-bdc0-11f45296dcba-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.513803 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.513777 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j"] Nov 01 07:09:29.567315 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.567278 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt"] Nov 01 07:09:29.607001 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.606969 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4311f1-4db2-447e-a999-89d431c64f59-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.607001 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.607002 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.607245 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.607027 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm2b2\" (UniqueName: \"kubernetes.io/projected/29951118-24dc-49e4-bdc0-11f45296dcba-kube-api-access-jm2b2\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.607245 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.607220 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4311f1-4db2-447e-a999-89d431c64f59-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.607245 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.607243 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h7gk\" (UniqueName: \"kubernetes.io/projected/8d4311f1-4db2-447e-a999-89d431c64f59-kube-api-access-5h7gk\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.607410 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.607267 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.607410 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.607287 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29951118-24dc-49e4-bdc0-11f45296dcba-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.607541 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.607517 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.607604 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.607562 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29951118-24dc-49e4-bdc0-11f45296dcba-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.607664 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.607603 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.607664 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.607609 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.607664 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.607639 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.607905 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.607669 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4311f1-4db2-447e-a999-89d431c64f59-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.607905 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.607754 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.607905 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.607796 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.607905 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.607828 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29951118-24dc-49e4-bdc0-11f45296dcba-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.607905 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.607869 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.607905 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.607871 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.608231 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.607926 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.608231 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.607961 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.608231 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.608008 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.608392 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.608325 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.608450 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.608386 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29951118-24dc-49e4-bdc0-11f45296dcba-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.610155 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.610128 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.610789 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.610767 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29951118-24dc-49e4-bdc0-11f45296dcba-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.663508 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.663448 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29951118-24dc-49e4-bdc0-11f45296dcba-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.663630 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.663559 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm2b2\" (UniqueName: \"kubernetes.io/projected/29951118-24dc-49e4-bdc0-11f45296dcba-kube-api-access-jm2b2\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.673137 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.673109 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:29.709060 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.709031 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4311f1-4db2-447e-a999-89d431c64f59-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.709060 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.709068 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.709319 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.709120 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4311f1-4db2-447e-a999-89d431c64f59-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.709319 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.709147 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h7gk\" (UniqueName: \"kubernetes.io/projected/8d4311f1-4db2-447e-a999-89d431c64f59-kube-api-access-5h7gk\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.709319 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.709210 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.709319 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.709237 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4311f1-4db2-447e-a999-89d431c64f59-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.709674 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.709644 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.709931 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.709823 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.709931 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.709263 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.709931 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.709915 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.709931 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.709937 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.709931 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.709980 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.710427 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.710201 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4311f1-4db2-447e-a999-89d431c64f59-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.710427 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.710331 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.712102 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.712076 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.712102 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.712095 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4311f1-4db2-447e-a999-89d431c64f59-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.723210 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.723190 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:29.766464 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.766435 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h7gk\" (UniqueName: \"kubernetes.io/projected/8d4311f1-4db2-447e-a999-89d431c64f59-kube-api-access-5h7gk\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.766618 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.766567 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4311f1-4db2-447e-a999-89d431c64f59-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:29.772872 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:29.772790 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:30.150427 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:30.150404 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j"] Nov 01 07:09:30.151401 ip-10-0-140-68 kubenswrapper[4385]: W1101 07:09:30.151375 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29951118_24dc_49e4_bdc0_11f45296dcba.slice/crio-54c6fae3864d57b002bcd3310872d0baf169930135a13b512066761b60d1d76e WatchSource:0}: Error finding container 54c6fae3864d57b002bcd3310872d0baf169930135a13b512066761b60d1d76e: Status 404 returned error can't find the container with id 54c6fae3864d57b002bcd3310872d0baf169930135a13b512066761b60d1d76e Nov 01 07:09:30.157374 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:30.157338 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:09:30.157474 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:30.157422 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:09:30.157474 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:30.157467 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:09:30.203350 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:30.203323 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt"] Nov 01 07:09:30.203691 ip-10-0-140-68 kubenswrapper[4385]: W1101 07:09:30.203650 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4311f1_4db2_447e_a999_89d431c64f59.slice/crio-7c50722978c926a42781e20032b3bcee2047687c5b1763201a5b156c4cc5d7f9 WatchSource:0}: Error finding container 7c50722978c926a42781e20032b3bcee2047687c5b1763201a5b156c4cc5d7f9: Status 404 returned error can't find the container with id 7c50722978c926a42781e20032b3bcee2047687c5b1763201a5b156c4cc5d7f9 Nov 01 07:09:30.206257 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:30.206223 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:09:30.206336 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:30.206320 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:09:30.206382 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:30.206368 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:09:30.747317 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:30.747287 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" event={"ID":"29951118-24dc-49e4-bdc0-11f45296dcba","Type":"ContainerStarted","Data":"2da73ebc14738e27eae5aa30b80c94260d5f41b285ed57e2a85bc097bf08a35a"} Nov 01 07:09:30.747805 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:30.747339 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" event={"ID":"29951118-24dc-49e4-bdc0-11f45296dcba","Type":"ContainerStarted","Data":"54c6fae3864d57b002bcd3310872d0baf169930135a13b512066761b60d1d76e"} Nov 01 07:09:30.749245 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:30.749217 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" event={"ID":"8d4311f1-4db2-447e-a999-89d431c64f59","Type":"ContainerStarted","Data":"91a6c1d51f7fc66231568d3af3047d681368a3633308ceb9e3e576d23a84e2f9"} Nov 01 07:09:30.749337 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:30.749275 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" event={"ID":"8d4311f1-4db2-447e-a999-89d431c64f59","Type":"ContainerStarted","Data":"7c50722978c926a42781e20032b3bcee2047687c5b1763201a5b156c4cc5d7f9"} Nov 01 07:09:30.773322 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:30.773291 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:30.866998 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:30.866934 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" podStartSLOduration=1.866909157 podStartE2EDuration="1.866909157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:09:30.862887592 +0000 UTC m=+3580.626687450" watchObservedRunningTime="2026-11-01 07:09:30.866909157 +0000 UTC m=+3580.630709018" Nov 01 07:09:30.983081 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:30.983023 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" podStartSLOduration=1.983008364 podStartE2EDuration="1.983008364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:09:30.979996703 +0000 UTC m=+3580.743796559" watchObservedRunningTime="2026-11-01 07:09:30.983008364 +0000 UTC m=+3580.746808221" Nov 01 07:09:31.108951 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:31.108755 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:31.253445 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:31.253402 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:31.638593 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:31.638544 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:31.725643 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:31.725609 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:31.732021 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:31.731985 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 01 07:09:31.732273 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:31.732243 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" podUID="29951118-24dc-49e4-bdc0-11f45296dcba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 01 07:09:31.797031 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:31.796991 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 01 07:09:31.798688 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:31.798648 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" podUID="8d4311f1-4db2-447e-a999-89d431c64f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 01 07:09:32.651074 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:32.651037 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:32.728868 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:32.724220 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 01 07:09:32.728868 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:32.724296 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" podUID="29951118-24dc-49e4-bdc0-11f45296dcba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 01 07:09:32.789228 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:32.775310 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 01 07:09:32.789228 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:32.775367 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" podUID="8d4311f1-4db2-447e-a999-89d431c64f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 01 07:09:32.821860 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:32.821322 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:33.724859 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:33.724362 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 01 07:09:33.724859 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:33.724440 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" podUID="29951118-24dc-49e4-bdc0-11f45296dcba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 01 07:09:33.774316 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:33.774183 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 01 07:09:33.774316 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:33.774255 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" podUID="8d4311f1-4db2-447e-a999-89d431c64f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 01 07:09:34.311859 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:34.311490 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:34.724086 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:34.724050 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 01 07:09:34.724365 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:34.724335 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" podUID="29951118-24dc-49e4-bdc0-11f45296dcba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 01 07:09:34.782863 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:34.782797 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 01 07:09:34.783188 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:34.783120 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" podUID="8d4311f1-4db2-447e-a999-89d431c64f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 01 07:09:35.725185 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:35.724709 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 01 07:09:35.725185 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:35.724769 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" podUID="29951118-24dc-49e4-bdc0-11f45296dcba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 01 07:09:35.774392 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:35.774259 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 01 07:09:35.774392 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:35.774318 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" podUID="8d4311f1-4db2-447e-a999-89d431c64f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 01 07:09:36.194901 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:36.194189 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:36.726259 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:36.726226 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 01 07:09:36.726675 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:36.726285 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" podUID="29951118-24dc-49e4-bdc0-11f45296dcba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 01 07:09:36.774348 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:36.774314 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 01 07:09:36.774644 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:36.774586 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" podUID="8d4311f1-4db2-447e-a999-89d431c64f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 01 07:09:36.803611 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:36.801252 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:36.827261 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:09:36.827218 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:09:37.036684 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:37.036530 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:37.220857 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:37.220723 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw"] Nov 01 07:09:37.221070 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:37.220983 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" podUID="ac1084eb-b2da-42cd-8376-f54f321c37b8" containerName="istio-proxy" containerID="cri-o://6f8fc53704ea10305d1efcfcd4e3badc98ca3287ea07cb0ce7afd1422cdc4900" gracePeriod=30 Nov 01 07:09:37.244537 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:37.244505 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:38.029434 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:38.029392 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:38.186852 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:38.186804 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:38.387028 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:38.386997 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:38.387906 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:38.387878 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:38.389568 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:38.389548 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:38.536051 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:38.536021 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:38.536462 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:38.536441 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:38.537320 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:38.537290 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:38.860425 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:38.860371 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:39.637127 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:39.637087 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:39.792202 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:39.792166 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:42.490617 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.490596 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 07:09:42.612159 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.612123 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-data\") pod \"ac1084eb-b2da-42cd-8376-f54f321c37b8\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " Nov 01 07:09:42.612159 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.612162 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-workload-certs\") pod \"ac1084eb-b2da-42cd-8376-f54f321c37b8\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " Nov 01 07:09:42.612427 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.612211 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-envoy\") pod \"ac1084eb-b2da-42cd-8376-f54f321c37b8\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " Nov 01 07:09:42.612427 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.612241 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-workload-socket\") pod \"ac1084eb-b2da-42cd-8376-f54f321c37b8\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " Nov 01 07:09:42.612427 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.612274 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-credential-socket\") pod \"ac1084eb-b2da-42cd-8376-f54f321c37b8\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " Nov 01 07:09:42.612427 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.612313 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac1084eb-b2da-42cd-8376-f54f321c37b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1084eb-b2da-42cd-8376-f54f321c37b8-istiod-ca-cert\") pod \"ac1084eb-b2da-42cd-8376-f54f321c37b8\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " Nov 01 07:09:42.612427 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.612357 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-podinfo\") pod \"ac1084eb-b2da-42cd-8376-f54f321c37b8\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " Nov 01 07:09:42.612427 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.612384 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-token\" (UniqueName: \"kubernetes.io/projected/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-token\") pod \"ac1084eb-b2da-42cd-8376-f54f321c37b8\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " Nov 01 07:09:42.612759 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.612461 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-data" pod "ac1084eb-b2da-42cd-8376-f54f321c37b8" (UID: "ac1084eb-b2da-42cd-8376-f54f321c37b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:42.612759 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.612597 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-workload-socket" pod "ac1084eb-b2da-42cd-8376-f54f321c37b8" (UID: "ac1084eb-b2da-42cd-8376-f54f321c37b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:42.612759 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.612650 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-credential-socket" pod "ac1084eb-b2da-42cd-8376-f54f321c37b8" (UID: "ac1084eb-b2da-42cd-8376-f54f321c37b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:42.612759 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.612726 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac1084eb-b2da-42cd-8376-f54f321c37b8-kube-api-access-9tnsl\" (UniqueName: \"kubernetes.io/projected/ac1084eb-b2da-42cd-8376-f54f321c37b8-kube-api-access-9tnsl\") pod \"ac1084eb-b2da-42cd-8376-f54f321c37b8\" (UID: \"ac1084eb-b2da-42cd-8376-f54f321c37b8\") " Nov 01 07:09:42.612998 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.612856 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac1084eb-b2da-42cd-8376-f54f321c37b8-istiod-ca-cert" pod "ac1084eb-b2da-42cd-8376-f54f321c37b8" (UID: "ac1084eb-b2da-42cd-8376-f54f321c37b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:09:42.613269 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.612903 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-workload-certs" pod "ac1084eb-b2da-42cd-8376-f54f321c37b8" (UID: "ac1084eb-b2da-42cd-8376-f54f321c37b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:42.613975 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.613942 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:42.625267 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.625200 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:42.625462 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.625446 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:42.625603 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.625565 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1084eb-b2da-42cd-8376-f54f321c37b8-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:42.625746 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.618692 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-envoy" pod "ac1084eb-b2da-42cd-8376-f54f321c37b8" (UID: "ac1084eb-b2da-42cd-8376-f54f321c37b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:42.625899 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.625524 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-token" pod "ac1084eb-b2da-42cd-8376-f54f321c37b8" (UID: "ac1084eb-b2da-42cd-8376-f54f321c37b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:42.628748 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.626570 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-podinfo" pod "ac1084eb-b2da-42cd-8376-f54f321c37b8" (UID: "ac1084eb-b2da-42cd-8376-f54f321c37b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:09:42.628748 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.628689 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1084eb-b2da-42cd-8376-f54f321c37b8-kube-api-access-9tnsl" pod "ac1084eb-b2da-42cd-8376-f54f321c37b8" (UID: "ac1084eb-b2da-42cd-8376-f54f321c37b8"). InnerVolumeSpecName "kube-api-access-9tnsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:42.726876 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.726826 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tnsl\" (UniqueName: \"kubernetes.io/projected/ac1084eb-b2da-42cd-8376-f54f321c37b8-kube-api-access-9tnsl\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:42.726876 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.726873 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:42.726876 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.726883 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:42.727123 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.726894 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:42.727123 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.726902 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1084eb-b2da-42cd-8376-f54f321c37b8-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:42.871173 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.871086 4385 generic.go:356] "Generic (PLEG): container finished" podID="ac1084eb-b2da-42cd-8376-f54f321c37b8" containerID="6f8fc53704ea10305d1efcfcd4e3badc98ca3287ea07cb0ce7afd1422cdc4900" exitCode=0 Nov 01 07:09:42.871173 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.871148 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" Nov 01 07:09:42.871360 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.871178 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" event={"ID":"ac1084eb-b2da-42cd-8376-f54f321c37b8","Type":"ContainerDied","Data":"6f8fc53704ea10305d1efcfcd4e3badc98ca3287ea07cb0ce7afd1422cdc4900"} Nov 01 07:09:42.871360 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.871260 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw" event={"ID":"ac1084eb-b2da-42cd-8376-f54f321c37b8","Type":"ContainerDied","Data":"98920fe1e3d12bbceabc8b5ea11292b664d5983755a5fa4ad1b72c9212510226"} Nov 01 07:09:42.871360 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.871284 4385 scope.go:122] "RemoveContainer" containerID="6f8fc53704ea10305d1efcfcd4e3badc98ca3287ea07cb0ce7afd1422cdc4900" Nov 01 07:09:42.882225 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.882208 4385 scope.go:122] "RemoveContainer" containerID="6f8fc53704ea10305d1efcfcd4e3badc98ca3287ea07cb0ce7afd1422cdc4900" Nov 01 07:09:42.882503 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:09:42.882478 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f8fc53704ea10305d1efcfcd4e3badc98ca3287ea07cb0ce7afd1422cdc4900\": container with ID starting with 6f8fc53704ea10305d1efcfcd4e3badc98ca3287ea07cb0ce7afd1422cdc4900 not found: ID does not exist" containerID="6f8fc53704ea10305d1efcfcd4e3badc98ca3287ea07cb0ce7afd1422cdc4900" Nov 01 07:09:42.882605 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:42.882511 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f8fc53704ea10305d1efcfcd4e3badc98ca3287ea07cb0ce7afd1422cdc4900"} err="failed to get container status \"6f8fc53704ea10305d1efcfcd4e3badc98ca3287ea07cb0ce7afd1422cdc4900\": rpc error: code = NotFound desc = could not find container \"6f8fc53704ea10305d1efcfcd4e3badc98ca3287ea07cb0ce7afd1422cdc4900\": container with ID starting with 6f8fc53704ea10305d1efcfcd4e3badc98ca3287ea07cb0ce7afd1422cdc4900 not found: ID does not exist" Nov 01 07:09:43.034970 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:43.034939 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw"] Nov 01 07:09:43.085970 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:43.085933 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-gw9sw"] Nov 01 07:09:43.733663 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:43.733624 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:44.386822 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:44.386785 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac1084eb-b2da-42cd-8376-f54f321c37b8" path="/var/lib/kubelet/pods/ac1084eb-b2da-42cd-8376-f54f321c37b8/volumes" Nov 01 07:09:45.944864 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:45.943649 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:46.175885 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:46.175640 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:47.717852 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:47.717805 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:48.199264 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:48.199136 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j"] Nov 01 07:09:48.199473 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:48.199377 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" podUID="29951118-24dc-49e4-bdc0-11f45296dcba" containerName="istio-proxy" containerID="cri-o://2da73ebc14738e27eae5aa30b80c94260d5f41b285ed57e2a85bc097bf08a35a" gracePeriod=30 Nov 01 07:09:48.217206 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:48.217167 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt"] Nov 01 07:09:48.217427 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:48.217399 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" podUID="8d4311f1-4db2-447e-a999-89d431c64f59" containerName="istio-proxy" containerID="cri-o://91a6c1d51f7fc66231568d3af3047d681368a3633308ceb9e3e576d23a84e2f9" gracePeriod=30 Nov 01 07:09:48.219234 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:48.219183 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:48.234077 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:48.223880 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b"] Nov 01 07:09:48.234077 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:48.224108 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" podUID="ceea7f64-b7d4-4954-a474-9c80d954cab9" containerName="istio-proxy" containerID="cri-o://8768329706d1936d36ccebe4378cc624156fe5e70412d98edba86177aff7e8ee" gracePeriod=30 Nov 01 07:09:48.957906 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:48.957867 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:51.233671 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:51.233641 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 07:09:51.234089 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:51.233690 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" podUID="ceea7f64-b7d4-4954-a474-9c80d954cab9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 07:09:53.391111 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.391056 4385 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 01 07:09:53.391111 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.391143 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" podUID="8d4311f1-4db2-447e-a999-89d431c64f59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 01 07:09:53.513495 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.513474 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:53.535453 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.535423 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:53.568138 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.568114 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:09:53.573785 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.573754 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-credential-socket\") pod \"8d4311f1-4db2-447e-a999-89d431c64f59\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " Nov 01 07:09:53.573899 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.573816 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d4311f1-4db2-447e-a999-89d431c64f59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4311f1-4db2-447e-a999-89d431c64f59-istiod-ca-cert\") pod \"8d4311f1-4db2-447e-a999-89d431c64f59\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " Nov 01 07:09:53.573955 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.573901 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d4311f1-4db2-447e-a999-89d431c64f59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4311f1-4db2-447e-a999-89d431c64f59-istio-podinfo\") pod \"8d4311f1-4db2-447e-a999-89d431c64f59\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " Nov 01 07:09:53.574016 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.573954 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-workload-socket\") pod \"8d4311f1-4db2-447e-a999-89d431c64f59\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " Nov 01 07:09:53.574072 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.574022 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-credential-socket" pod "8d4311f1-4db2-447e-a999-89d431c64f59" (UID: "8d4311f1-4db2-447e-a999-89d431c64f59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.574072 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.574027 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d4311f1-4db2-447e-a999-89d431c64f59-istio-token\" (UniqueName: \"kubernetes.io/projected/8d4311f1-4db2-447e-a999-89d431c64f59-istio-token\") pod \"8d4311f1-4db2-447e-a999-89d431c64f59\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " Nov 01 07:09:53.574188 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.574135 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-workload-certs\") pod \"8d4311f1-4db2-447e-a999-89d431c64f59\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " Nov 01 07:09:53.574250 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.574189 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-istio-data\") pod \"8d4311f1-4db2-447e-a999-89d431c64f59\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " Nov 01 07:09:53.574309 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.574237 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d4311f1-4db2-447e-a999-89d431c64f59-istiod-ca-cert" pod "8d4311f1-4db2-447e-a999-89d431c64f59" (UID: "8d4311f1-4db2-447e-a999-89d431c64f59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:09:53.574309 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.574280 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d4311f1-4db2-447e-a999-89d431c64f59-kube-api-access-5h7gk\" (UniqueName: \"kubernetes.io/projected/8d4311f1-4db2-447e-a999-89d431c64f59-kube-api-access-5h7gk\") pod \"8d4311f1-4db2-447e-a999-89d431c64f59\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " Nov 01 07:09:53.574433 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.574314 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-istio-envoy\") pod \"8d4311f1-4db2-447e-a999-89d431c64f59\" (UID: \"8d4311f1-4db2-447e-a999-89d431c64f59\") " Nov 01 07:09:53.574433 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.574419 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-workload-certs" pod "8d4311f1-4db2-447e-a999-89d431c64f59" (UID: "8d4311f1-4db2-447e-a999-89d431c64f59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.574548 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.574459 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-istio-data" pod "8d4311f1-4db2-447e-a999-89d431c64f59" (UID: "8d4311f1-4db2-447e-a999-89d431c64f59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.574763 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.574709 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.574763 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.574736 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.574763 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.574752 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.574763 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.574765 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4311f1-4db2-447e-a999-89d431c64f59-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.575031 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.574710 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-workload-socket" pod "8d4311f1-4db2-447e-a999-89d431c64f59" (UID: "8d4311f1-4db2-447e-a999-89d431c64f59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.576173 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.576149 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d4311f1-4db2-447e-a999-89d431c64f59-istio-podinfo" pod "8d4311f1-4db2-447e-a999-89d431c64f59" (UID: "8d4311f1-4db2-447e-a999-89d431c64f59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:09:53.576509 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.576489 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4311f1-4db2-447e-a999-89d431c64f59-istio-token" pod "8d4311f1-4db2-447e-a999-89d431c64f59" (UID: "8d4311f1-4db2-447e-a999-89d431c64f59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:53.576902 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.576882 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4311f1-4db2-447e-a999-89d431c64f59-kube-api-access-5h7gk" pod "8d4311f1-4db2-447e-a999-89d431c64f59" (UID: "8d4311f1-4db2-447e-a999-89d431c64f59"). InnerVolumeSpecName "kube-api-access-5h7gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:53.576990 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.576899 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-istio-envoy" pod "8d4311f1-4db2-447e-a999-89d431c64f59" (UID: "8d4311f1-4db2-447e-a999-89d431c64f59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.675690 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.675652 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-istio-envoy\") pod \"29951118-24dc-49e4-bdc0-11f45296dcba\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " Nov 01 07:09:53.675904 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.675701 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-envoy\") pod \"ceea7f64-b7d4-4954-a474-9c80d954cab9\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " Nov 01 07:09:53.675904 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.675734 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29951118-24dc-49e4-bdc0-11f45296dcba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29951118-24dc-49e4-bdc0-11f45296dcba-istio-podinfo\") pod \"29951118-24dc-49e4-bdc0-11f45296dcba\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " Nov 01 07:09:53.675904 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.675759 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceea7f64-b7d4-4954-a474-9c80d954cab9-kube-api-access-v8v7k\" (UniqueName: \"kubernetes.io/projected/ceea7f64-b7d4-4954-a474-9c80d954cab9-kube-api-access-v8v7k\") pod \"ceea7f64-b7d4-4954-a474-9c80d954cab9\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " Nov 01 07:09:53.675904 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.675785 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-podinfo\") pod \"ceea7f64-b7d4-4954-a474-9c80d954cab9\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " Nov 01 07:09:53.675904 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.675809 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-credential-socket\") pod \"ceea7f64-b7d4-4954-a474-9c80d954cab9\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " Nov 01 07:09:53.676280 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.676063 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-token\" (UniqueName: \"kubernetes.io/projected/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-token\") pod \"ceea7f64-b7d4-4954-a474-9c80d954cab9\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " Nov 01 07:09:53.676280 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.676157 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-credential-socket\") pod \"29951118-24dc-49e4-bdc0-11f45296dcba\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " Nov 01 07:09:53.676280 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.676209 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29951118-24dc-49e4-bdc0-11f45296dcba-istio-token\" (UniqueName: \"kubernetes.io/projected/29951118-24dc-49e4-bdc0-11f45296dcba-istio-token\") pod \"29951118-24dc-49e4-bdc0-11f45296dcba\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " Nov 01 07:09:53.676280 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.676261 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29951118-24dc-49e4-bdc0-11f45296dcba-kube-api-access-jm2b2\" (UniqueName: \"kubernetes.io/projected/29951118-24dc-49e4-bdc0-11f45296dcba-kube-api-access-jm2b2\") pod \"29951118-24dc-49e4-bdc0-11f45296dcba\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " Nov 01 07:09:53.676503 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.676317 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-workload-socket\") pod \"29951118-24dc-49e4-bdc0-11f45296dcba\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " Nov 01 07:09:53.676503 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.676365 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-data\") pod \"ceea7f64-b7d4-4954-a474-9c80d954cab9\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " Nov 01 07:09:53.678727 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.676156 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-credential-socket" pod "ceea7f64-b7d4-4954-a474-9c80d954cab9" (UID: "ceea7f64-b7d4-4954-a474-9c80d954cab9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.678922 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.676346 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-credential-socket" pod "29951118-24dc-49e4-bdc0-11f45296dcba" (UID: "29951118-24dc-49e4-bdc0-11f45296dcba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.678922 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.676584 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-data" pod "ceea7f64-b7d4-4954-a474-9c80d954cab9" (UID: "ceea7f64-b7d4-4954-a474-9c80d954cab9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.678922 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.678688 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceea7f64-b7d4-4954-a474-9c80d954cab9-kube-api-access-v8v7k" pod "ceea7f64-b7d4-4954-a474-9c80d954cab9" (UID: "ceea7f64-b7d4-4954-a474-9c80d954cab9"). InnerVolumeSpecName "kube-api-access-v8v7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:53.678922 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.678686 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-envoy" pod "ceea7f64-b7d4-4954-a474-9c80d954cab9" (UID: "ceea7f64-b7d4-4954-a474-9c80d954cab9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.678922 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.678794 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-workload-certs\") pod \"29951118-24dc-49e4-bdc0-11f45296dcba\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " Nov 01 07:09:53.678922 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.678903 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-workload-certs\") pod \"ceea7f64-b7d4-4954-a474-9c80d954cab9\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " Nov 01 07:09:53.679277 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.678926 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-workload-socket" pod "29951118-24dc-49e4-bdc0-11f45296dcba" (UID: "29951118-24dc-49e4-bdc0-11f45296dcba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.679277 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.678951 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-istio-data\") pod \"29951118-24dc-49e4-bdc0-11f45296dcba\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " Nov 01 07:09:53.679277 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.678985 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-workload-socket\") pod \"ceea7f64-b7d4-4954-a474-9c80d954cab9\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " Nov 01 07:09:53.679277 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679032 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceea7f64-b7d4-4954-a474-9c80d954cab9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceea7f64-b7d4-4954-a474-9c80d954cab9-istiod-ca-cert\") pod \"ceea7f64-b7d4-4954-a474-9c80d954cab9\" (UID: \"ceea7f64-b7d4-4954-a474-9c80d954cab9\") " Nov 01 07:09:53.679277 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679042 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29951118-24dc-49e4-bdc0-11f45296dcba-istio-podinfo" pod "29951118-24dc-49e4-bdc0-11f45296dcba" (UID: "29951118-24dc-49e4-bdc0-11f45296dcba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:09:53.679277 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679067 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29951118-24dc-49e4-bdc0-11f45296dcba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29951118-24dc-49e4-bdc0-11f45296dcba-istiod-ca-cert\") pod \"29951118-24dc-49e4-bdc0-11f45296dcba\" (UID: \"29951118-24dc-49e4-bdc0-11f45296dcba\") " Nov 01 07:09:53.679277 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679068 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29951118-24dc-49e4-bdc0-11f45296dcba-istio-token" pod "29951118-24dc-49e4-bdc0-11f45296dcba" (UID: "29951118-24dc-49e4-bdc0-11f45296dcba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:53.679277 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679152 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-workload-certs" pod "29951118-24dc-49e4-bdc0-11f45296dcba" (UID: "29951118-24dc-49e4-bdc0-11f45296dcba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.679697 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679283 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29951118-24dc-49e4-bdc0-11f45296dcba-kube-api-access-jm2b2" pod "29951118-24dc-49e4-bdc0-11f45296dcba" (UID: "29951118-24dc-49e4-bdc0-11f45296dcba"). InnerVolumeSpecName "kube-api-access-jm2b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:53.679697 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679296 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-istio-data" pod "29951118-24dc-49e4-bdc0-11f45296dcba" (UID: "29951118-24dc-49e4-bdc0-11f45296dcba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.679697 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679330 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-workload-certs" pod "ceea7f64-b7d4-4954-a474-9c80d954cab9" (UID: "ceea7f64-b7d4-4954-a474-9c80d954cab9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.679697 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679389 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-workload-socket" pod "ceea7f64-b7d4-4954-a474-9c80d954cab9" (UID: "ceea7f64-b7d4-4954-a474-9c80d954cab9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.679697 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679548 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4311f1-4db2-447e-a999-89d431c64f59-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.679697 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679578 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.679697 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679593 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.679697 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679607 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29951118-24dc-49e4-bdc0-11f45296dcba-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.679697 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679617 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceea7f64-b7d4-4954-a474-9c80d954cab9-istiod-ca-cert" pod "ceea7f64-b7d4-4954-a474-9c80d954cab9" (UID: "ceea7f64-b7d4-4954-a474-9c80d954cab9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:09:53.679697 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679620 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4311f1-4db2-447e-a999-89d431c64f59-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.679697 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679638 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29951118-24dc-49e4-bdc0-11f45296dcba-istiod-ca-cert" pod "29951118-24dc-49e4-bdc0-11f45296dcba" (UID: "29951118-24dc-49e4-bdc0-11f45296dcba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:09:53.679697 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679648 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-istio-envoy" pod "29951118-24dc-49e4-bdc0-11f45296dcba" (UID: "29951118-24dc-49e4-bdc0-11f45296dcba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.679697 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679665 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8v7k\" (UniqueName: \"kubernetes.io/projected/ceea7f64-b7d4-4954-a474-9c80d954cab9-kube-api-access-v8v7k\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.679697 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679683 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.679697 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679697 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.680237 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679712 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29951118-24dc-49e4-bdc0-11f45296dcba-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.680237 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679725 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm2b2\" (UniqueName: \"kubernetes.io/projected/29951118-24dc-49e4-bdc0-11f45296dcba-kube-api-access-jm2b2\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.680237 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679740 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.680237 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679747 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-podinfo" pod "ceea7f64-b7d4-4954-a474-9c80d954cab9" (UID: "ceea7f64-b7d4-4954-a474-9c80d954cab9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:09:53.680237 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679755 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.680237 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679770 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.680237 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679785 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.680237 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679798 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.680237 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679813 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h7gk\" (UniqueName: \"kubernetes.io/projected/8d4311f1-4db2-447e-a999-89d431c64f59-kube-api-access-5h7gk\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.680237 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679848 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4311f1-4db2-447e-a999-89d431c64f59-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.680237 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.679865 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceea7f64-b7d4-4954-a474-9c80d954cab9-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.680802 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.680785 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-token" pod "ceea7f64-b7d4-4954-a474-9c80d954cab9" (UID: "ceea7f64-b7d4-4954-a474-9c80d954cab9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:53.780501 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.780424 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.780501 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.780458 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceea7f64-b7d4-4954-a474-9c80d954cab9-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.780501 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.780472 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceea7f64-b7d4-4954-a474-9c80d954cab9-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.780501 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.780485 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29951118-24dc-49e4-bdc0-11f45296dcba-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.780501 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.780498 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29951118-24dc-49e4-bdc0-11f45296dcba-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.929767 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.929730 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:53.983353 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.983316 4385 generic.go:356] "Generic (PLEG): container finished" podID="8d4311f1-4db2-447e-a999-89d431c64f59" containerID="91a6c1d51f7fc66231568d3af3047d681368a3633308ceb9e3e576d23a84e2f9" exitCode=0 Nov 01 07:09:53.983550 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.983375 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" Nov 01 07:09:53.983550 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.983404 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" event={"ID":"8d4311f1-4db2-447e-a999-89d431c64f59","Type":"ContainerDied","Data":"91a6c1d51f7fc66231568d3af3047d681368a3633308ceb9e3e576d23a84e2f9"} Nov 01 07:09:53.983550 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.983495 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt" event={"ID":"8d4311f1-4db2-447e-a999-89d431c64f59","Type":"ContainerDied","Data":"7c50722978c926a42781e20032b3bcee2047687c5b1763201a5b156c4cc5d7f9"} Nov 01 07:09:53.983550 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.983525 4385 scope.go:122] "RemoveContainer" containerID="91a6c1d51f7fc66231568d3af3047d681368a3633308ceb9e3e576d23a84e2f9" Nov 01 07:09:53.985194 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.985164 4385 generic.go:356] "Generic (PLEG): container finished" podID="ceea7f64-b7d4-4954-a474-9c80d954cab9" containerID="8768329706d1936d36ccebe4378cc624156fe5e70412d98edba86177aff7e8ee" exitCode=0 Nov 01 07:09:53.985324 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.985232 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" Nov 01 07:09:53.985324 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.985252 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" event={"ID":"ceea7f64-b7d4-4954-a474-9c80d954cab9","Type":"ContainerDied","Data":"8768329706d1936d36ccebe4378cc624156fe5e70412d98edba86177aff7e8ee"} Nov 01 07:09:53.985324 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.985311 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b" event={"ID":"ceea7f64-b7d4-4954-a474-9c80d954cab9","Type":"ContainerDied","Data":"a63b9b533209f800435b9ba571fff143d1431f801763858a70b2272578856db2"} Nov 01 07:09:53.986805 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.986783 4385 generic.go:356] "Generic (PLEG): container finished" podID="29951118-24dc-49e4-bdc0-11f45296dcba" containerID="2da73ebc14738e27eae5aa30b80c94260d5f41b285ed57e2a85bc097bf08a35a" exitCode=0 Nov 01 07:09:53.986942 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.986860 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" Nov 01 07:09:53.986942 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.986861 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" event={"ID":"29951118-24dc-49e4-bdc0-11f45296dcba","Type":"ContainerDied","Data":"2da73ebc14738e27eae5aa30b80c94260d5f41b285ed57e2a85bc097bf08a35a"} Nov 01 07:09:53.986942 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.986902 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j" event={"ID":"29951118-24dc-49e4-bdc0-11f45296dcba","Type":"ContainerDied","Data":"54c6fae3864d57b002bcd3310872d0baf169930135a13b512066761b60d1d76e"} Nov 01 07:09:53.996268 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.996238 4385 scope.go:122] "RemoveContainer" containerID="91a6c1d51f7fc66231568d3af3047d681368a3633308ceb9e3e576d23a84e2f9" Nov 01 07:09:53.996610 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:09:53.996548 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a6c1d51f7fc66231568d3af3047d681368a3633308ceb9e3e576d23a84e2f9\": container with ID starting with 91a6c1d51f7fc66231568d3af3047d681368a3633308ceb9e3e576d23a84e2f9 not found: ID does not exist" containerID="91a6c1d51f7fc66231568d3af3047d681368a3633308ceb9e3e576d23a84e2f9" Nov 01 07:09:53.996610 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.996579 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a6c1d51f7fc66231568d3af3047d681368a3633308ceb9e3e576d23a84e2f9"} err="failed to get container status \"91a6c1d51f7fc66231568d3af3047d681368a3633308ceb9e3e576d23a84e2f9\": rpc error: code = NotFound desc = could not find container \"91a6c1d51f7fc66231568d3af3047d681368a3633308ceb9e3e576d23a84e2f9\": container with ID starting with 91a6c1d51f7fc66231568d3af3047d681368a3633308ceb9e3e576d23a84e2f9 not found: ID does not exist" Nov 01 07:09:53.996610 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:53.996603 4385 scope.go:122] "RemoveContainer" containerID="8768329706d1936d36ccebe4378cc624156fe5e70412d98edba86177aff7e8ee" Nov 01 07:09:54.005784 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:54.005769 4385 scope.go:122] "RemoveContainer" containerID="8768329706d1936d36ccebe4378cc624156fe5e70412d98edba86177aff7e8ee" Nov 01 07:09:54.006118 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:09:54.006097 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8768329706d1936d36ccebe4378cc624156fe5e70412d98edba86177aff7e8ee\": container with ID starting with 8768329706d1936d36ccebe4378cc624156fe5e70412d98edba86177aff7e8ee not found: ID does not exist" containerID="8768329706d1936d36ccebe4378cc624156fe5e70412d98edba86177aff7e8ee" Nov 01 07:09:54.006211 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:54.006128 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8768329706d1936d36ccebe4378cc624156fe5e70412d98edba86177aff7e8ee"} err="failed to get container status \"8768329706d1936d36ccebe4378cc624156fe5e70412d98edba86177aff7e8ee\": rpc error: code = NotFound desc = could not find container \"8768329706d1936d36ccebe4378cc624156fe5e70412d98edba86177aff7e8ee\": container with ID starting with 8768329706d1936d36ccebe4378cc624156fe5e70412d98edba86177aff7e8ee not found: ID does not exist" Nov 01 07:09:54.006211 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:54.006148 4385 scope.go:122] "RemoveContainer" containerID="2da73ebc14738e27eae5aa30b80c94260d5f41b285ed57e2a85bc097bf08a35a" Nov 01 07:09:54.017988 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:54.017963 4385 scope.go:122] "RemoveContainer" containerID="2da73ebc14738e27eae5aa30b80c94260d5f41b285ed57e2a85bc097bf08a35a" Nov 01 07:09:54.018472 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:09:54.018453 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da73ebc14738e27eae5aa30b80c94260d5f41b285ed57e2a85bc097bf08a35a\": container with ID starting with 2da73ebc14738e27eae5aa30b80c94260d5f41b285ed57e2a85bc097bf08a35a not found: ID does not exist" containerID="2da73ebc14738e27eae5aa30b80c94260d5f41b285ed57e2a85bc097bf08a35a" Nov 01 07:09:54.018544 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:54.018479 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da73ebc14738e27eae5aa30b80c94260d5f41b285ed57e2a85bc097bf08a35a"} err="failed to get container status \"2da73ebc14738e27eae5aa30b80c94260d5f41b285ed57e2a85bc097bf08a35a\": rpc error: code = NotFound desc = could not find container \"2da73ebc14738e27eae5aa30b80c94260d5f41b285ed57e2a85bc097bf08a35a\": container with ID starting with 2da73ebc14738e27eae5aa30b80c94260d5f41b285ed57e2a85bc097bf08a35a not found: ID does not exist" Nov 01 07:09:54.149500 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:54.149463 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j"] Nov 01 07:09:54.153228 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:54.153198 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-v768j"] Nov 01 07:09:54.261356 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:54.261324 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt"] Nov 01 07:09:54.267887 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:54.267823 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-9r5pt"] Nov 01 07:09:54.375176 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:54.375145 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b"] Nov 01 07:09:54.385090 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:54.385059 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29951118-24dc-49e4-bdc0-11f45296dcba" path="/var/lib/kubelet/pods/29951118-24dc-49e4-bdc0-11f45296dcba/volumes" Nov 01 07:09:54.385633 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:54.385613 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d4311f1-4db2-447e-a999-89d431c64f59" path="/var/lib/kubelet/pods/8d4311f1-4db2-447e-a999-89d431c64f59/volumes" Nov 01 07:09:54.386033 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:54.386016 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-tjn2b"] Nov 01 07:09:55.205066 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:55.205031 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:09:56.383587 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:56.383552 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceea7f64-b7d4-4954-a474-9c80d954cab9" path="/var/lib/kubelet/pods/ceea7f64-b7d4-4954-a474-9c80d954cab9/volumes" Nov 01 07:09:58.610215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:09:58.610182 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:10:03.261258 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:10:03.261223 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:10:06.777988 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:10:06.777954 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:10:08.024411 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:10:08.024370 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:10:12.765986 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:10:12.765951 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:10:17.427441 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:10:17.427403 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:10:22.117174 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:10:22.117137 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:10:26.829292 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:10:26.829254 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:10:31.516557 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:10:31.516466 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:10:36.227786 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:10:36.227752 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:10:36.779122 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:10:36.779078 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:10:40.916570 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:10:40.916538 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:10:43.748886 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:10:43.748854 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:10:43.892784 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:10:43.892752 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:10:45.616316 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:10:45.616288 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:10:50.320151 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:10:50.320117 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:10:54.497149 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:10:54.497110 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:10:55.015370 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:10:55.015331 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:10:59.718254 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:10:59.718221 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:00.175710 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:00.175676 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:00.325699 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:00.325661 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:04.532299 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:04.532263 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:06.778746 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:11:06.778710 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:11:09.223893 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:09.223861 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:11.159471 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:11.159432 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:11.457701 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:11.457614 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:13.929768 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:13.929733 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:18.702957 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:18.702915 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:23.442920 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:23.442890 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:25.203899 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:25.203858 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:28.126561 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:28.126527 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:32.631577 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:32.631540 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:32.827559 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:32.827529 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:36.779065 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:11:36.779030 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:11:36.790356 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:36.790327 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:37.535116 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:37.535082 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:38.496375 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:38.496331 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:42.228363 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:42.228328 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:46.919572 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:46.919522 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:51.622579 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:51.622540 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:53.495515 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:53.495484 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:54.829084 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:54.829055 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:11:56.331169 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:11:56.331133 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:12:01.017116 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:12:01.017074 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:12:05.727452 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:12:05.727410 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:12:06.779151 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:12:06.779111 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:12:10.424253 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:12:10.424220 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:12:15.119113 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:12:15.119073 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:12:19.911346 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:12:19.911306 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:12:24.620985 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:12:24.620947 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:12:29.320888 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:12:29.320851 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:12:34.015137 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:12:34.015102 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:12:36.779565 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:12:36.779528 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:12:38.715224 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:12:38.715186 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:12:43.413605 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:12:43.413562 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:12:48.146557 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:12:48.146522 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:12:52.826434 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:12:52.826393 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:12:55.173259 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:12:55.173224 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:12:55.310285 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:12:55.310254 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:12:57.515925 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:12:57.515885 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:02.209080 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:02.209045 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:03.107493 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:03.107455 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:06.778876 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:13:06.778821 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:13:06.928531 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:06.928490 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:08.146070 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:08.146027 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:09.645473 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:09.645431 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:11.629814 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:11.629777 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:14.938718 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:14.938678 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2"] Nov 01 07:13:15.047144 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:15.047113 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb"] Nov 01 07:13:15.322878 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:15.322753 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:15.537364 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:15.537329 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:15.685362 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:15.685316 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:16.678541 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:16.678504 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:19.939096 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:19.939040 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2" podUID="d00aff18-fddd-4938-aa74-87917e5a721d" containerName="payload-processing" containerID="cri-o://16acb300fa4028f4d4a214ac77c3831ed98c8f669b0de1666d9264c01c4aeab7" gracePeriod=25 Nov 01 07:13:20.048223 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:20.048164 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb" podUID="b0f1c27f-e62f-42ed-8633-829b8b66e8d0" containerName="payload-pre-processing" containerID="cri-o://1c7f5d387e902703b205d57f79de606e0115a9e251bce9c4be21057e30195f55" gracePeriod=25 Nov 01 07:13:20.113939 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:20.113897 4385 generic.go:356] "Generic (PLEG): container finished" podID="d00aff18-fddd-4938-aa74-87917e5a721d" containerID="16acb300fa4028f4d4a214ac77c3831ed98c8f669b0de1666d9264c01c4aeab7" exitCode=0 Nov 01 07:13:20.114145 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:20.114075 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2" event={"ID":"d00aff18-fddd-4938-aa74-87917e5a721d","Type":"ContainerDied","Data":"16acb300fa4028f4d4a214ac77c3831ed98c8f669b0de1666d9264c01c4aeab7"} Nov 01 07:13:20.220790 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:20.220766 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2" Nov 01 07:13:20.340099 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:20.340068 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb" Nov 01 07:13:20.367152 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:20.367123 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d00aff18-fddd-4938-aa74-87917e5a721d-kube-api-access-hfxl4\" (UniqueName: \"kubernetes.io/projected/d00aff18-fddd-4938-aa74-87917e5a721d-kube-api-access-hfxl4\") pod \"d00aff18-fddd-4938-aa74-87917e5a721d\" (UID: \"d00aff18-fddd-4938-aa74-87917e5a721d\") " Nov 01 07:13:20.367364 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:20.367164 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d00aff18-fddd-4938-aa74-87917e5a721d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d00aff18-fddd-4938-aa74-87917e5a721d-plugins-config-volume\") pod \"d00aff18-fddd-4938-aa74-87917e5a721d\" (UID: \"d00aff18-fddd-4938-aa74-87917e5a721d\") " Nov 01 07:13:20.367810 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:20.367780 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d00aff18-fddd-4938-aa74-87917e5a721d-plugins-config-volume" pod "d00aff18-fddd-4938-aa74-87917e5a721d" (UID: "d00aff18-fddd-4938-aa74-87917e5a721d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:13:20.370132 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:20.370106 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00aff18-fddd-4938-aa74-87917e5a721d-kube-api-access-hfxl4" pod "d00aff18-fddd-4938-aa74-87917e5a721d" (UID: "d00aff18-fddd-4938-aa74-87917e5a721d"). InnerVolumeSpecName "kube-api-access-hfxl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:13:20.468048 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:20.467965 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0f1c27f-e62f-42ed-8633-829b8b66e8d0-kube-api-access-4hthj\" (UniqueName: \"kubernetes.io/projected/b0f1c27f-e62f-42ed-8633-829b8b66e8d0-kube-api-access-4hthj\") pod \"b0f1c27f-e62f-42ed-8633-829b8b66e8d0\" (UID: \"b0f1c27f-e62f-42ed-8633-829b8b66e8d0\") " Nov 01 07:13:20.468215 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:20.468082 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0f1c27f-e62f-42ed-8633-829b8b66e8d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f1c27f-e62f-42ed-8633-829b8b66e8d0-plugins-config-volume\") pod \"b0f1c27f-e62f-42ed-8633-829b8b66e8d0\" (UID: \"b0f1c27f-e62f-42ed-8633-829b8b66e8d0\") " Nov 01 07:13:20.468464 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:20.468441 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfxl4\" (UniqueName: \"kubernetes.io/projected/d00aff18-fddd-4938-aa74-87917e5a721d-kube-api-access-hfxl4\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:13:20.468590 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:20.468468 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d00aff18-fddd-4938-aa74-87917e5a721d-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:13:20.468590 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:20.468482 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0f1c27f-e62f-42ed-8633-829b8b66e8d0-plugins-config-volume" pod "b0f1c27f-e62f-42ed-8633-829b8b66e8d0" (UID: "b0f1c27f-e62f-42ed-8633-829b8b66e8d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:13:20.470184 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:20.470162 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f1c27f-e62f-42ed-8633-829b8b66e8d0-kube-api-access-4hthj" pod "b0f1c27f-e62f-42ed-8633-829b8b66e8d0" (UID: "b0f1c27f-e62f-42ed-8633-829b8b66e8d0"). InnerVolumeSpecName "kube-api-access-4hthj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:13:20.570013 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:20.569979 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hthj\" (UniqueName: \"kubernetes.io/projected/b0f1c27f-e62f-42ed-8633-829b8b66e8d0-kube-api-access-4hthj\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:13:20.570013 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:20.570016 4385 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f1c27f-e62f-42ed-8633-829b8b66e8d0-plugins-config-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:13:21.122554 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:21.122514 4385 generic.go:356] "Generic (PLEG): container finished" podID="b0f1c27f-e62f-42ed-8633-829b8b66e8d0" containerID="1c7f5d387e902703b205d57f79de606e0115a9e251bce9c4be21057e30195f55" exitCode=0 Nov 01 07:13:21.123009 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:21.122558 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb" Nov 01 07:13:21.123009 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:21.122606 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb" event={"ID":"b0f1c27f-e62f-42ed-8633-829b8b66e8d0","Type":"ContainerDied","Data":"1c7f5d387e902703b205d57f79de606e0115a9e251bce9c4be21057e30195f55"} Nov 01 07:13:21.123009 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:21.122657 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb" event={"ID":"b0f1c27f-e62f-42ed-8633-829b8b66e8d0","Type":"ContainerDied","Data":"bd34c950abea8e1864c1db6855c17b32b8a1c048c270ff3743fb2e6ec7ea5b46"} Nov 01 07:13:21.123009 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:21.122680 4385 scope.go:122] "RemoveContainer" containerID="1c7f5d387e902703b205d57f79de606e0115a9e251bce9c4be21057e30195f55" Nov 01 07:13:21.124194 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:21.124173 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2" event={"ID":"d00aff18-fddd-4938-aa74-87917e5a721d","Type":"ContainerDied","Data":"77099b7ac559989c62666c445f99904e6d353a19e1b2b49bf6b64c951a60cac5"} Nov 01 07:13:21.124277 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:21.124263 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2" Nov 01 07:13:21.134748 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:21.134730 4385 scope.go:122] "RemoveContainer" containerID="1c7f5d387e902703b205d57f79de606e0115a9e251bce9c4be21057e30195f55" Nov 01 07:13:21.135040 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:13:21.135019 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c7f5d387e902703b205d57f79de606e0115a9e251bce9c4be21057e30195f55\": container with ID starting with 1c7f5d387e902703b205d57f79de606e0115a9e251bce9c4be21057e30195f55 not found: ID does not exist" containerID="1c7f5d387e902703b205d57f79de606e0115a9e251bce9c4be21057e30195f55" Nov 01 07:13:21.135135 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:21.135054 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c7f5d387e902703b205d57f79de606e0115a9e251bce9c4be21057e30195f55"} err="failed to get container status \"1c7f5d387e902703b205d57f79de606e0115a9e251bce9c4be21057e30195f55\": rpc error: code = NotFound desc = could not find container \"1c7f5d387e902703b205d57f79de606e0115a9e251bce9c4be21057e30195f55\": container with ID starting with 1c7f5d387e902703b205d57f79de606e0115a9e251bce9c4be21057e30195f55 not found: ID does not exist" Nov 01 07:13:21.135135 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:21.135079 4385 scope.go:122] "RemoveContainer" containerID="16acb300fa4028f4d4a214ac77c3831ed98c8f669b0de1666d9264c01c4aeab7" Nov 01 07:13:21.291604 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:21.291525 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb"] Nov 01 07:13:21.341339 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:21.341293 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-28c598-69f558c87f-h7kvb"] Nov 01 07:13:21.415632 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:21.415598 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2"] Nov 01 07:13:21.425617 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:21.425583 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-28c598-55548bdb55-j92v2"] Nov 01 07:13:21.683979 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:21.683944 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:22.382892 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:22.382860 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0f1c27f-e62f-42ed-8633-829b8b66e8d0" path="/var/lib/kubelet/pods/b0f1c27f-e62f-42ed-8633-829b8b66e8d0/volumes" Nov 01 07:13:22.383289 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:22.383228 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d00aff18-fddd-4938-aa74-87917e5a721d" path="/var/lib/kubelet/pods/d00aff18-fddd-4938-aa74-87917e5a721d/volumes" Nov 01 07:13:22.437077 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:22.437040 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:26.413312 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:26.413275 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:27.880253 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:27.880213 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:30.437791 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:30.437756 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9"] Nov 01 07:13:30.438290 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:30.438001 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" podUID="c2bfe801-8222-497c-85ce-1fc5695f32e7" containerName="maas-api" containerID="cri-o://a747680a3558aa280511876d75e87c5084ebf57c6a62e1c232a8d40bd606ec5e" gracePeriod=30 Nov 01 07:13:30.558695 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:30.558220 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:30.718665 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:30.718644 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" Nov 01 07:13:30.869118 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:30.869083 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2bfe801-8222-497c-85ce-1fc5695f32e7-kube-api-access-5dgvc\" (UniqueName: \"kubernetes.io/projected/c2bfe801-8222-497c-85ce-1fc5695f32e7-kube-api-access-5dgvc\") pod \"c2bfe801-8222-497c-85ce-1fc5695f32e7\" (UID: \"c2bfe801-8222-497c-85ce-1fc5695f32e7\") " Nov 01 07:13:30.869318 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:30.869132 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2bfe801-8222-497c-85ce-1fc5695f32e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2bfe801-8222-497c-85ce-1fc5695f32e7-metrics-certs\") pod \"c2bfe801-8222-497c-85ce-1fc5695f32e7\" (UID: \"c2bfe801-8222-497c-85ce-1fc5695f32e7\") " Nov 01 07:13:30.869318 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:30.869165 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2bfe801-8222-497c-85ce-1fc5695f32e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bfe801-8222-497c-85ce-1fc5695f32e7-openshift-service-ca\") pod \"c2bfe801-8222-497c-85ce-1fc5695f32e7\" (UID: \"c2bfe801-8222-497c-85ce-1fc5695f32e7\") " Nov 01 07:13:30.869318 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:30.869195 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2bfe801-8222-497c-85ce-1fc5695f32e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2bfe801-8222-497c-85ce-1fc5695f32e7-maas-api-tls\") pod \"c2bfe801-8222-497c-85ce-1fc5695f32e7\" (UID: \"c2bfe801-8222-497c-85ce-1fc5695f32e7\") " Nov 01 07:13:30.869633 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:30.869598 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2bfe801-8222-497c-85ce-1fc5695f32e7-openshift-service-ca" pod "c2bfe801-8222-497c-85ce-1fc5695f32e7" (UID: "c2bfe801-8222-497c-85ce-1fc5695f32e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:13:30.871570 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:30.871538 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2bfe801-8222-497c-85ce-1fc5695f32e7-metrics-certs" pod "c2bfe801-8222-497c-85ce-1fc5695f32e7" (UID: "c2bfe801-8222-497c-85ce-1fc5695f32e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:13:30.871698 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:30.871673 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2bfe801-8222-497c-85ce-1fc5695f32e7-kube-api-access-5dgvc" pod "c2bfe801-8222-497c-85ce-1fc5695f32e7" (UID: "c2bfe801-8222-497c-85ce-1fc5695f32e7"). InnerVolumeSpecName "kube-api-access-5dgvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:13:30.871891 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:30.871871 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2bfe801-8222-497c-85ce-1fc5695f32e7-maas-api-tls" pod "c2bfe801-8222-497c-85ce-1fc5695f32e7" (UID: "c2bfe801-8222-497c-85ce-1fc5695f32e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:13:30.971053 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:30.970946 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dgvc\" (UniqueName: \"kubernetes.io/projected/c2bfe801-8222-497c-85ce-1fc5695f32e7-kube-api-access-5dgvc\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:13:30.971053 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:30.970988 4385 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2bfe801-8222-497c-85ce-1fc5695f32e7-metrics-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:13:30.971053 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:30.971005 4385 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bfe801-8222-497c-85ce-1fc5695f32e7-openshift-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:13:30.971053 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:30.971018 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2bfe801-8222-497c-85ce-1fc5695f32e7-maas-api-tls\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:13:31.183813 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:31.183781 4385 generic.go:356] "Generic (PLEG): container finished" podID="c2bfe801-8222-497c-85ce-1fc5695f32e7" containerID="a747680a3558aa280511876d75e87c5084ebf57c6a62e1c232a8d40bd606ec5e" exitCode=0 Nov 01 07:13:31.184004 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:31.183850 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" Nov 01 07:13:31.184004 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:31.183857 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" event={"ID":"c2bfe801-8222-497c-85ce-1fc5695f32e7","Type":"ContainerDied","Data":"a747680a3558aa280511876d75e87c5084ebf57c6a62e1c232a8d40bd606ec5e"} Nov 01 07:13:31.184004 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:31.183905 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9" event={"ID":"c2bfe801-8222-497c-85ce-1fc5695f32e7","Type":"ContainerDied","Data":"9da0e001c407c5451da0d843929adc726b8b69bc30f3aa01b55fcdec1b4bcadb"} Nov 01 07:13:31.184004 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:31.183927 4385 scope.go:122] "RemoveContainer" containerID="a747680a3558aa280511876d75e87c5084ebf57c6a62e1c232a8d40bd606ec5e" Nov 01 07:13:31.194408 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:31.194386 4385 scope.go:122] "RemoveContainer" containerID="a747680a3558aa280511876d75e87c5084ebf57c6a62e1c232a8d40bd606ec5e" Nov 01 07:13:31.194705 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:13:31.194680 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a747680a3558aa280511876d75e87c5084ebf57c6a62e1c232a8d40bd606ec5e\": container with ID starting with a747680a3558aa280511876d75e87c5084ebf57c6a62e1c232a8d40bd606ec5e not found: ID does not exist" containerID="a747680a3558aa280511876d75e87c5084ebf57c6a62e1c232a8d40bd606ec5e" Nov 01 07:13:31.194791 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:31.194719 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a747680a3558aa280511876d75e87c5084ebf57c6a62e1c232a8d40bd606ec5e"} err="failed to get container status \"a747680a3558aa280511876d75e87c5084ebf57c6a62e1c232a8d40bd606ec5e\": rpc error: code = NotFound desc = could not find container \"a747680a3558aa280511876d75e87c5084ebf57c6a62e1c232a8d40bd606ec5e\": container with ID starting with a747680a3558aa280511876d75e87c5084ebf57c6a62e1c232a8d40bd606ec5e not found: ID does not exist" Nov 01 07:13:31.357860 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:31.351783 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9"] Nov 01 07:13:31.357860 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:31.357559 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-28c598-7c78cbd4d-krds9"] Nov 01 07:13:32.318845 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:32.318791 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:32.383878 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:32.383847 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2bfe801-8222-497c-85ce-1fc5695f32e7" path="/var/lib/kubelet/pods/c2bfe801-8222-497c-85ce-1fc5695f32e7/volumes" Nov 01 07:13:33.373241 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:33.373207 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:33.524217 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:33.524183 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:36.779553 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:13:36.779509 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:13:37.032193 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:37.032088 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:41.723386 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:41.723351 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:44.573814 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:44.573745 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:44.725383 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:44.725343 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:45.166652 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:45.166597 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:46.501321 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:46.501283 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:47.775798 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:47.775757 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:48.410564 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:48.410530 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:49.158171 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:49.158129 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:50.667642 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:50.667605 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:50.865788 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:50.865746 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:51.643879 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:51.643843 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:55.047308 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:55.047262 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:55.504074 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:55.504039 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:56.095461 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:56.095426 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:56.336484 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:56.336447 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:56.497406 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:56.497326 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:56.755381 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:56.755294 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:13:56.906929 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:13:56.906894 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:00.498251 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:00.498215 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:00.813693 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:00.813607 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:02.523897 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:02.523821 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:06.295876 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:06.295811 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:06.598379 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:06.598337 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:06.779189 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:14:06.779145 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:14:07.288351 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:07.288314 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:07.858388 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:07.858350 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:08.372270 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:08.372237 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:09.115425 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:09.115384 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:12.557779 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:12.557745 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:15.073223 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:15.073185 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:16.096000 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:16.095964 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:16.250801 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:16.250766 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:17.420575 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:17.420537 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:20.590043 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:20.589995 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:21.750582 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:21.750539 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:26.426357 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:26.426325 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:26.595458 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:26.595418 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:27.940410 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:27.940375 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:28.408890 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:28.408856 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:29.304183 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:29.304156 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58588b67c6-mr9nn_413fadb6-87d9-4270-b53e-a7bf64b32d60/authorino/0.log" Nov 01 07:14:29.721961 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:29.721932 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-6jbbb_1203942c-3f66-4dbc-b88a-7ab1ea9154be/manager/0.log" Nov 01 07:14:29.947070 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:29.947042 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-5hrk6_d6a699b6-b7e6-4e66-8a91-2db8f6209e11/manager/0.log" Nov 01 07:14:30.164101 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:30.164070 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pwpvc_045ea95f-c9de-4c1e-af56-ff3712203d6a/kuadrant-console-plugin/0.log" Nov 01 07:14:31.302196 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:31.302157 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-jpshj_2eb5e419-7160-44b3-b69a-5f9fc85b6609/manager/0.log" Nov 01 07:14:31.918216 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:31.918185 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-q7mrd_79830bd7-36da-4336-86d4-8e63d1a7e440/manager/0.log" Nov 01 07:14:33.333516 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:33.333478 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-k9s7l_34d1d6b5-e7a8-43b0-ba57-4ea70b9d1169/manager/0.log" Nov 01 07:14:33.935109 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:33.935072 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-446vb_57957a77-1fea-4f3d-a619-a655a33498be/manager/0.log" Nov 01 07:14:35.182651 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:35.182619 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-4nlhs_9b76cbc0-abbe-45e8-a061-e44e87c2f782/manager/0.log" Nov 01 07:14:36.779549 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:14:36.779507 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:14:40.395532 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:40.395494 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:40.996242 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:40.996208 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:41.377891 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:41.377828 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:45.550518 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:45.550474 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:45.753255 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:45.753216 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:48.744898 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:48.744857 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:48.924363 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:48.924315 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:51.877640 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:51.877593 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:52.041478 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:52.041448 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:52.189688 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:52.189594 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:55.861947 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:55.861906 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:14:58.211265 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:14:58.211234 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:15:00.155658 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.155624 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk"] Nov 01 07:15:00.156060 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.156048 4385 cpu_manager.go:414] "containerMap: removing container" podUID="d00aff18-fddd-4938-aa74-87917e5a721d" containerName="payload-processing" Nov 01 07:15:00.156118 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.156060 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d00aff18-fddd-4938-aa74-87917e5a721d" containerName="payload-processing" Nov 01 07:15:00.156118 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.156069 4385 cpu_manager.go:414] "containerMap: removing container" podUID="ac1084eb-b2da-42cd-8376-f54f321c37b8" containerName="istio-proxy" Nov 01 07:15:00.156118 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.156074 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac1084eb-b2da-42cd-8376-f54f321c37b8" containerName="istio-proxy" Nov 01 07:15:00.156118 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.156086 4385 cpu_manager.go:414] "containerMap: removing container" podUID="ceea7f64-b7d4-4954-a474-9c80d954cab9" containerName="istio-proxy" Nov 01 07:15:00.156118 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.156091 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceea7f64-b7d4-4954-a474-9c80d954cab9" containerName="istio-proxy" Nov 01 07:15:00.156118 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.156101 4385 cpu_manager.go:414] "containerMap: removing container" podUID="c2bfe801-8222-497c-85ce-1fc5695f32e7" containerName="maas-api" Nov 01 07:15:00.156118 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.156107 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2bfe801-8222-497c-85ce-1fc5695f32e7" containerName="maas-api" Nov 01 07:15:00.156118 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.156116 4385 cpu_manager.go:414] "containerMap: removing container" podUID="29951118-24dc-49e4-bdc0-11f45296dcba" containerName="istio-proxy" Nov 01 07:15:00.156118 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.156121 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29951118-24dc-49e4-bdc0-11f45296dcba" containerName="istio-proxy" Nov 01 07:15:00.156441 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.156126 4385 cpu_manager.go:414] "containerMap: removing container" podUID="8d4311f1-4db2-447e-a999-89d431c64f59" containerName="istio-proxy" Nov 01 07:15:00.156441 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.156131 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d4311f1-4db2-447e-a999-89d431c64f59" containerName="istio-proxy" Nov 01 07:15:00.156441 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.156144 4385 cpu_manager.go:414] "containerMap: removing container" podUID="b0f1c27f-e62f-42ed-8633-829b8b66e8d0" containerName="payload-pre-processing" Nov 01 07:15:00.156441 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.156148 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0f1c27f-e62f-42ed-8633-829b8b66e8d0" containerName="payload-pre-processing" Nov 01 07:15:00.156441 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.156207 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceea7f64-b7d4-4954-a474-9c80d954cab9" containerName="istio-proxy" Nov 01 07:15:00.156441 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.156215 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2bfe801-8222-497c-85ce-1fc5695f32e7" containerName="maas-api" Nov 01 07:15:00.156441 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.156223 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="29951118-24dc-49e4-bdc0-11f45296dcba" containerName="istio-proxy" Nov 01 07:15:00.156441 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.156228 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d4311f1-4db2-447e-a999-89d431c64f59" containerName="istio-proxy" Nov 01 07:15:00.156441 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.156233 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0f1c27f-e62f-42ed-8633-829b8b66e8d0" containerName="payload-pre-processing" Nov 01 07:15:00.156441 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.156242 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="d00aff18-fddd-4938-aa74-87917e5a721d" containerName="payload-processing" Nov 01 07:15:00.156441 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.156247 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac1084eb-b2da-42cd-8376-f54f321c37b8" containerName="istio-proxy" Nov 01 07:15:00.159092 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.159076 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk" Nov 01 07:15:00.217697 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.217660 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk"] Nov 01 07:15:00.262335 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.262297 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42ggl\"" Nov 01 07:15:00.293602 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.293574 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3802f6-96b7-4b16-9696-da0eeee659ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk\" (UID: \"9a3802f6-96b7-4b16-9696-da0eeee659ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk" Nov 01 07:15:00.293782 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.293636 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwpr6\" (UniqueName: \"kubernetes.io/projected/9a3802f6-96b7-4b16-9696-da0eeee659ef-kube-api-access-dwpr6\") pod \"maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk\" (UID: \"9a3802f6-96b7-4b16-9696-da0eeee659ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk" Nov 01 07:15:00.394623 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.394580 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3802f6-96b7-4b16-9696-da0eeee659ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk\" (UID: \"9a3802f6-96b7-4b16-9696-da0eeee659ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk" Nov 01 07:15:00.394856 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.394701 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwpr6\" (UniqueName: \"kubernetes.io/projected/9a3802f6-96b7-4b16-9696-da0eeee659ef-kube-api-access-dwpr6\") pod \"maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk\" (UID: \"9a3802f6-96b7-4b16-9696-da0eeee659ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk" Nov 01 07:15:00.396044 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.396015 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3802f6-96b7-4b16-9696-da0eeee659ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk\" (UID: \"9a3802f6-96b7-4b16-9696-da0eeee659ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk" Nov 01 07:15:00.451378 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.451295 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwpr6\" (UniqueName: \"kubernetes.io/projected/9a3802f6-96b7-4b16-9696-da0eeee659ef-kube-api-access-dwpr6\") pod \"maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk\" (UID: \"9a3802f6-96b7-4b16-9696-da0eeee659ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk" Nov 01 07:15:00.473160 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.473128 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk" Nov 01 07:15:00.843713 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:00.843684 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk"] Nov 01 07:15:00.845213 ip-10-0-140-68 kubenswrapper[4385]: W1101 07:15:00.845188 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a3802f6_96b7_4b16_9696_da0eeee659ef.slice/crio-e124d067383a4da8fac67f9a00b16d714b6147ef36f83c764e229a77c3572dfa WatchSource:0}: Error finding container e124d067383a4da8fac67f9a00b16d714b6147ef36f83c764e229a77c3572dfa: Status 404 returned error can't find the container with id e124d067383a4da8fac67f9a00b16d714b6147ef36f83c764e229a77c3572dfa Nov 01 07:15:01.518656 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:01.518545 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:15:01.696076 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:01.696048 4385 generic.go:356] "Generic (PLEG): container finished" podID="9a3802f6-96b7-4b16-9696-da0eeee659ef" containerID="090d9a10c67136978347a816c64d4e5ee3d44066904bd041b9b6384ebafcea0d" exitCode=60 Nov 01 07:15:01.696197 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:01.696142 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk" event={"ID":"9a3802f6-96b7-4b16-9696-da0eeee659ef","Type":"ContainerDied","Data":"090d9a10c67136978347a816c64d4e5ee3d44066904bd041b9b6384ebafcea0d"} Nov 01 07:15:01.696197 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:01.696177 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk" event={"ID":"9a3802f6-96b7-4b16-9696-da0eeee659ef","Type":"ContainerStarted","Data":"e124d067383a4da8fac67f9a00b16d714b6147ef36f83c764e229a77c3572dfa"} Nov 01 07:15:01.696333 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:01.696318 4385 scope.go:122] "RemoveContainer" containerID="090d9a10c67136978347a816c64d4e5ee3d44066904bd041b9b6384ebafcea0d" Nov 01 07:15:02.703443 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:02.703408 4385 generic.go:356] "Generic (PLEG): container finished" podID="9a3802f6-96b7-4b16-9696-da0eeee659ef" containerID="56690a208f93093bfd3091f168c0981d899b6681773226cb84bf9ef1d7edd8ef" exitCode=60 Nov 01 07:15:02.703947 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:02.703498 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk" event={"ID":"9a3802f6-96b7-4b16-9696-da0eeee659ef","Type":"ContainerDied","Data":"56690a208f93093bfd3091f168c0981d899b6681773226cb84bf9ef1d7edd8ef"} Nov 01 07:15:02.703947 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:02.703564 4385 scope.go:122] "RemoveContainer" containerID="090d9a10c67136978347a816c64d4e5ee3d44066904bd041b9b6384ebafcea0d" Nov 01 07:15:02.703947 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:02.703873 4385 scope.go:122] "RemoveContainer" containerID="56690a208f93093bfd3091f168c0981d899b6681773226cb84bf9ef1d7edd8ef" Nov 01 07:15:02.704176 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:15:02.704154 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk_odh-ai-gateway-infra(9a3802f6-96b7-4b16-9696-da0eeee659ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk" podUID="9a3802f6-96b7-4b16-9696-da0eeee659ef" Nov 01 07:15:03.709884 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:03.709861 4385 scope.go:122] "RemoveContainer" containerID="56690a208f93093bfd3091f168c0981d899b6681773226cb84bf9ef1d7edd8ef" Nov 01 07:15:03.710285 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:15:03.710080 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk_odh-ai-gateway-infra(9a3802f6-96b7-4b16-9696-da0eeee659ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk" podUID="9a3802f6-96b7-4b16-9696-da0eeee659ef" Nov 01 07:15:06.222375 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:06.222333 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:15:06.778286 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:15:06.778254 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:15:10.412078 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:10.412044 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:15:11.026627 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:11.026588 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:15:11.234434 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:11.234394 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:15:11.391732 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:11.391676 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:15:13.377461 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:13.377433 4385 scope.go:122] "RemoveContainer" containerID="56690a208f93093bfd3091f168c0981d899b6681773226cb84bf9ef1d7edd8ef" Nov 01 07:15:13.768398 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:13.768363 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk" event={"ID":"9a3802f6-96b7-4b16-9696-da0eeee659ef","Type":"ContainerStarted","Data":"022823ab86ad061db8fe0057a13586696761c714c8ea9d318e8df462d057d7b8"} Nov 01 07:15:13.882754 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:13.882706 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk" podStartSLOduration=13.882692853 podStartE2EDuration="13.882692853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:15:13.881128191 +0000 UTC m=+3923.644928046" watchObservedRunningTime="2026-11-01 07:15:13.882692853 +0000 UTC m=+3923.646492710" Nov 01 07:15:14.775991 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.775961 4385 generic.go:356] "Generic (PLEG): container finished" podID="9a3802f6-96b7-4b16-9696-da0eeee659ef" containerID="022823ab86ad061db8fe0057a13586696761c714c8ea9d318e8df462d057d7b8" exitCode=60 Nov 01 07:15:14.776494 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.776027 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk" event={"ID":"9a3802f6-96b7-4b16-9696-da0eeee659ef","Type":"ContainerDied","Data":"022823ab86ad061db8fe0057a13586696761c714c8ea9d318e8df462d057d7b8"} Nov 01 07:15:14.776494 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.776092 4385 scope.go:122] "RemoveContainer" containerID="56690a208f93093bfd3091f168c0981d899b6681773226cb84bf9ef1d7edd8ef" Nov 01 07:15:14.776494 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.776424 4385 scope.go:122] "RemoveContainer" containerID="022823ab86ad061db8fe0057a13586696761c714c8ea9d318e8df462d057d7b8" Nov 01 07:15:14.776716 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:15:14.776694 4385 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk_odh-ai-gateway-infra(9a3802f6-96b7-4b16-9696-da0eeee659ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk" podUID="9a3802f6-96b7-4b16-9696-da0eeee659ef" Nov 01 07:15:14.801135 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.801100 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9"] Nov 01 07:15:14.805556 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.805535 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:14.824123 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.824098 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-home\") pod \"simulator-main-976689-kserve-5c6db9ddfb-nxhv9\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:14.824241 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.824138 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db4341b8-4e10-4138-ae76-c4b4f8204a88-tls-certs\") pod \"simulator-main-976689-kserve-5c6db9ddfb-nxhv9\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:14.824332 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.824310 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-model-cache\") pod \"simulator-main-976689-kserve-5c6db9ddfb-nxhv9\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:14.824377 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.824356 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-tmp-dir\") pod \"simulator-main-976689-kserve-5c6db9ddfb-nxhv9\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:14.824442 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.824422 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfmdn\" (UniqueName: \"kubernetes.io/projected/db4341b8-4e10-4138-ae76-c4b4f8204a88-kube-api-access-qfmdn\") pod \"simulator-main-976689-kserve-5c6db9ddfb-nxhv9\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:14.824497 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.824482 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-dshm\") pod \"simulator-main-976689-kserve-5c6db9ddfb-nxhv9\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:14.862959 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.862930 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9"] Nov 01 07:15:14.877850 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.877803 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-main-976689\"/\"openshift-service-ca.crt\"" Nov 01 07:15:14.878185 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.878168 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-main-976689\"/\"default-dockercfg-zbj4g\"" Nov 01 07:15:14.878244 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.878174 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-main-976689\"/\"simulator-main-976689-kserve-self-signed-certs\"" Nov 01 07:15:14.897212 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.897184 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-main-976689\"/\"kube-root-ca.crt\"" Nov 01 07:15:14.911730 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.911699 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk"] Nov 01 07:15:14.925270 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.925245 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-model-cache\") pod \"simulator-main-976689-kserve-5c6db9ddfb-nxhv9\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:14.925410 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.925282 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-tmp-dir\") pod \"simulator-main-976689-kserve-5c6db9ddfb-nxhv9\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:14.925410 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.925305 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfmdn\" (UniqueName: \"kubernetes.io/projected/db4341b8-4e10-4138-ae76-c4b4f8204a88-kube-api-access-qfmdn\") pod \"simulator-main-976689-kserve-5c6db9ddfb-nxhv9\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:14.925410 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.925339 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-dshm\") pod \"simulator-main-976689-kserve-5c6db9ddfb-nxhv9\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:14.925410 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.925406 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-home\") pod \"simulator-main-976689-kserve-5c6db9ddfb-nxhv9\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:14.925626 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.925431 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db4341b8-4e10-4138-ae76-c4b4f8204a88-tls-certs\") pod \"simulator-main-976689-kserve-5c6db9ddfb-nxhv9\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:14.925695 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.925677 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-model-cache\") pod \"simulator-main-976689-kserve-5c6db9ddfb-nxhv9\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:14.925890 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.925856 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-tmp-dir\") pod \"simulator-main-976689-kserve-5c6db9ddfb-nxhv9\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:14.926035 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.925929 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-home\") pod \"simulator-main-976689-kserve-5c6db9ddfb-nxhv9\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:14.927904 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.927881 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-dshm\") pod \"simulator-main-976689-kserve-5c6db9ddfb-nxhv9\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:14.928461 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.928430 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db4341b8-4e10-4138-ae76-c4b4f8204a88-tls-certs\") pod \"simulator-main-976689-kserve-5c6db9ddfb-nxhv9\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:14.969279 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.969243 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:15:14.981531 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:14.981507 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfmdn\" (UniqueName: \"kubernetes.io/projected/db4341b8-4e10-4138-ae76-c4b4f8204a88-kube-api-access-qfmdn\") pod \"simulator-main-976689-kserve-5c6db9ddfb-nxhv9\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:15.117026 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:15.116988 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:15.493415 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:15.493387 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9"] Nov 01 07:15:15.494698 ip-10-0-140-68 kubenswrapper[4385]: W1101 07:15:15.494676 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb4341b8_4e10_4138_ae76_c4b4f8204a88.slice/crio-5eab586043399fa01137276a3b40f282decc722c57671c4ad4fe650e82b0229e WatchSource:0}: Error finding container 5eab586043399fa01137276a3b40f282decc722c57671c4ad4fe650e82b0229e: Status 404 returned error can't find the container with id 5eab586043399fa01137276a3b40f282decc722c57671c4ad4fe650e82b0229e Nov 01 07:15:15.788057 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:15.788027 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" event={"ID":"db4341b8-4e10-4138-ae76-c4b4f8204a88","Type":"ContainerStarted","Data":"dd9f6fe037a3b4219987f1654032efbbeb2ca24c2a70c275f897ebff873037c3"} Nov 01 07:15:15.788415 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:15.788077 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" event={"ID":"db4341b8-4e10-4138-ae76-c4b4f8204a88","Type":"ContainerStarted","Data":"5eab586043399fa01137276a3b40f282decc722c57671c4ad4fe650e82b0229e"} Nov 01 07:15:15.911684 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:15.911660 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk" Nov 01 07:15:15.935996 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:15.935975 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a3802f6-96b7-4b16-9696-da0eeee659ef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3802f6-96b7-4b16-9696-da0eeee659ef-maas-api-service-ca\") pod \"9a3802f6-96b7-4b16-9696-da0eeee659ef\" (UID: \"9a3802f6-96b7-4b16-9696-da0eeee659ef\") " Nov 01 07:15:15.936143 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:15.936049 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a3802f6-96b7-4b16-9696-da0eeee659ef-kube-api-access-dwpr6\" (UniqueName: \"kubernetes.io/projected/9a3802f6-96b7-4b16-9696-da0eeee659ef-kube-api-access-dwpr6\") pod \"9a3802f6-96b7-4b16-9696-da0eeee659ef\" (UID: \"9a3802f6-96b7-4b16-9696-da0eeee659ef\") " Nov 01 07:15:15.936468 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:15.936448 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a3802f6-96b7-4b16-9696-da0eeee659ef-maas-api-service-ca" pod "9a3802f6-96b7-4b16-9696-da0eeee659ef" (UID: "9a3802f6-96b7-4b16-9696-da0eeee659ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:15:15.938411 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:15.938389 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a3802f6-96b7-4b16-9696-da0eeee659ef-kube-api-access-dwpr6" pod "9a3802f6-96b7-4b16-9696-da0eeee659ef" (UID: "9a3802f6-96b7-4b16-9696-da0eeee659ef"). InnerVolumeSpecName "kube-api-access-dwpr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:15:16.015339 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:16.015280 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" podStartSLOduration=2.015266417 podStartE2EDuration="2.015266417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:15:16.01462427 +0000 UTC m=+3925.778424127" watchObservedRunningTime="2026-11-01 07:15:16.015266417 +0000 UTC m=+3925.779066273" Nov 01 07:15:16.033828 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:16.033770 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:15:16.037686 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:16.037637 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3802f6-96b7-4b16-9696-da0eeee659ef-maas-api-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:16.037686 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:16.037662 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwpr6\" (UniqueName: \"kubernetes.io/projected/9a3802f6-96b7-4b16-9696-da0eeee659ef-kube-api-access-dwpr6\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:16.118123 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:16.118098 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:16.796113 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:16.796022 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk" Nov 01 07:15:16.796113 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:16.796069 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk" event={"ID":"9a3802f6-96b7-4b16-9696-da0eeee659ef","Type":"ContainerDied","Data":"e124d067383a4da8fac67f9a00b16d714b6147ef36f83c764e229a77c3572dfa"} Nov 01 07:15:16.796596 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:16.796136 4385 scope.go:122] "RemoveContainer" containerID="022823ab86ad061db8fe0057a13586696761c714c8ea9d318e8df462d057d7b8" Nov 01 07:15:16.958653 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:16.958620 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk"] Nov 01 07:15:17.008490 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:17.008459 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-main-976689-29891955-dcrkk"] Nov 01 07:15:17.119979 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:17.119943 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:15:18.383620 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:18.383586 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a3802f6-96b7-4b16-9696-da0eeee659ef" path="/var/lib/kubelet/pods/9a3802f6-96b7-4b16-9696-da0eeee659ef/volumes" Nov 01 07:15:19.923426 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:19.923391 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9"] Nov 01 07:15:19.923889 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:19.923635 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" podUID="b74b50a6-08f2-4bc8-966a-1f23722f2807" containerName="istio-proxy" containerID="cri-o://758ad61d7e408860d9388cfdede12373201b5174f4c38d0c7cce905b138f4423" gracePeriod=30 Nov 01 07:15:20.808073 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:20.808031 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:15:25.118163 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.118127 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:25.130964 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.130936 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:25.171376 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.171346 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:15:25.232479 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.232407 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74b50a6-08f2-4bc8-966a-1f23722f2807-kube-api-access-79dv8\" (UniqueName: \"kubernetes.io/projected/b74b50a6-08f2-4bc8-966a-1f23722f2807-kube-api-access-79dv8\") pod \"b74b50a6-08f2-4bc8-966a-1f23722f2807\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " Nov 01 07:15:25.232479 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.232458 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-token\" (UniqueName: \"kubernetes.io/projected/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-token\") pod \"b74b50a6-08f2-4bc8-966a-1f23722f2807\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " Nov 01 07:15:25.232661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.232512 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-workload-certs\") pod \"b74b50a6-08f2-4bc8-966a-1f23722f2807\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " Nov 01 07:15:25.232661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.232541 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-data\") pod \"b74b50a6-08f2-4bc8-966a-1f23722f2807\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " Nov 01 07:15:25.232661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.232603 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-podinfo\") pod \"b74b50a6-08f2-4bc8-966a-1f23722f2807\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " Nov 01 07:15:25.232780 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.232683 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-credential-socket\") pod \"b74b50a6-08f2-4bc8-966a-1f23722f2807\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " Nov 01 07:15:25.232780 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.232715 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-workload-socket\") pod \"b74b50a6-08f2-4bc8-966a-1f23722f2807\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " Nov 01 07:15:25.232780 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.232743 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b74b50a6-08f2-4bc8-966a-1f23722f2807-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74b50a6-08f2-4bc8-966a-1f23722f2807-istiod-ca-cert\") pod \"b74b50a6-08f2-4bc8-966a-1f23722f2807\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " Nov 01 07:15:25.232968 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.232787 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-envoy\") pod \"b74b50a6-08f2-4bc8-966a-1f23722f2807\" (UID: \"b74b50a6-08f2-4bc8-966a-1f23722f2807\") " Nov 01 07:15:25.232968 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.232868 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-workload-certs" pod "b74b50a6-08f2-4bc8-966a-1f23722f2807" (UID: "b74b50a6-08f2-4bc8-966a-1f23722f2807"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:15:25.232968 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.232888 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-data" pod "b74b50a6-08f2-4bc8-966a-1f23722f2807" (UID: "b74b50a6-08f2-4bc8-966a-1f23722f2807"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:15:25.233134 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.233026 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-credential-socket" pod "b74b50a6-08f2-4bc8-966a-1f23722f2807" (UID: "b74b50a6-08f2-4bc8-966a-1f23722f2807"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:15:25.233134 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.233092 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-workload-socket" pod "b74b50a6-08f2-4bc8-966a-1f23722f2807" (UID: "b74b50a6-08f2-4bc8-966a-1f23722f2807"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:15:25.233248 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.233220 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b74b50a6-08f2-4bc8-966a-1f23722f2807-istiod-ca-cert" pod "b74b50a6-08f2-4bc8-966a-1f23722f2807" (UID: "b74b50a6-08f2-4bc8-966a-1f23722f2807"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:15:25.233311 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.233246 4385 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-credential-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.233311 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.233269 4385 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-workload-socket\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.233311 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.233284 4385 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-workload-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.233311 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.233299 4385 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-data\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.235008 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.234984 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74b50a6-08f2-4bc8-966a-1f23722f2807-kube-api-access-79dv8" pod "b74b50a6-08f2-4bc8-966a-1f23722f2807" (UID: "b74b50a6-08f2-4bc8-966a-1f23722f2807"). InnerVolumeSpecName "kube-api-access-79dv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:15:25.235126 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.235094 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-token" pod "b74b50a6-08f2-4bc8-966a-1f23722f2807" (UID: "b74b50a6-08f2-4bc8-966a-1f23722f2807"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:15:25.235485 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.235457 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-podinfo" pod "b74b50a6-08f2-4bc8-966a-1f23722f2807" (UID: "b74b50a6-08f2-4bc8-966a-1f23722f2807"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:15:25.235627 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.235528 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-envoy" pod "b74b50a6-08f2-4bc8-966a-1f23722f2807" (UID: "b74b50a6-08f2-4bc8-966a-1f23722f2807"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:15:25.334659 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.334628 4385 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-podinfo\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.334659 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.334654 4385 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74b50a6-08f2-4bc8-966a-1f23722f2807-istiod-ca-cert\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.334659 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.334665 4385 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-envoy\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.334924 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.334674 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79dv8\" (UniqueName: \"kubernetes.io/projected/b74b50a6-08f2-4bc8-966a-1f23722f2807-kube-api-access-79dv8\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.334924 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.334684 4385 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74b50a6-08f2-4bc8-966a-1f23722f2807-istio-token\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.568287 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.568203 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:15:25.849940 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.849901 4385 generic.go:356] "Generic (PLEG): container finished" podID="b74b50a6-08f2-4bc8-966a-1f23722f2807" containerID="758ad61d7e408860d9388cfdede12373201b5174f4c38d0c7cce905b138f4423" exitCode=0 Nov 01 07:15:25.850136 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.849974 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" Nov 01 07:15:25.850136 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.849985 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" event={"ID":"b74b50a6-08f2-4bc8-966a-1f23722f2807","Type":"ContainerDied","Data":"758ad61d7e408860d9388cfdede12373201b5174f4c38d0c7cce905b138f4423"} Nov 01 07:15:25.850136 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.850076 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9" event={"ID":"b74b50a6-08f2-4bc8-966a-1f23722f2807","Type":"ContainerDied","Data":"a3646831c225ccacaf5e62c0340a8875a49168344b5a6c182a08e0b47a35d05f"} Nov 01 07:15:25.850136 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.850101 4385 scope.go:122] "RemoveContainer" containerID="758ad61d7e408860d9388cfdede12373201b5174f4c38d0c7cce905b138f4423" Nov 01 07:15:25.860300 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.860283 4385 scope.go:122] "RemoveContainer" containerID="758ad61d7e408860d9388cfdede12373201b5174f4c38d0c7cce905b138f4423" Nov 01 07:15:25.860583 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:15:25.860564 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"758ad61d7e408860d9388cfdede12373201b5174f4c38d0c7cce905b138f4423\": container with ID starting with 758ad61d7e408860d9388cfdede12373201b5174f4c38d0c7cce905b138f4423 not found: ID does not exist" containerID="758ad61d7e408860d9388cfdede12373201b5174f4c38d0c7cce905b138f4423" Nov 01 07:15:25.860664 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.860592 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"758ad61d7e408860d9388cfdede12373201b5174f4c38d0c7cce905b138f4423"} err="failed to get container status \"758ad61d7e408860d9388cfdede12373201b5174f4c38d0c7cce905b138f4423\": rpc error: code = NotFound desc = could not find container \"758ad61d7e408860d9388cfdede12373201b5174f4c38d0c7cce905b138f4423\": container with ID starting with 758ad61d7e408860d9388cfdede12373201b5174f4c38d0c7cce905b138f4423 not found: ID does not exist" Nov 01 07:15:25.862401 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:25.862377 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:15:26.013622 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:26.013578 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9"] Nov 01 07:15:26.063068 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:26.063030 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8c8g9"] Nov 01 07:15:26.383659 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:26.383620 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b74b50a6-08f2-4bc8-966a-1f23722f2807" path="/var/lib/kubelet/pods/b74b50a6-08f2-4bc8-966a-1f23722f2807/volumes" Nov 01 07:15:30.324290 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:30.324240 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:15:35.022414 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:35.022380 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:15:36.778937 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:15:36.778898 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:15:39.729598 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:39.729537 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:15:40.785002 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:40.784960 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:15:44.542512 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:44.542473 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:15:49.233148 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:49.233114 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:15:53.937643 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:53.937608 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:15:56.165598 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:56.165554 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:15:56.328048 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:56.328005 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:15:58.646654 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:15:58.646621 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:16:03.343353 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:16:03.343307 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:16:06.779719 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:16:06.779678 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:16:08.029531 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:16:08.029500 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:16:12.720072 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:16:12.720033 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:16:17.417643 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:16:17.417606 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:16:22.125646 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:16:22.125602 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:16:26.843933 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:16:26.843898 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:16:36.780695 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:16:36.780601 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:16:56.511100 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:16:56.511060 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:17:01.224186 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:01.224145 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:17:02.616063 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:02.616030 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:17:02.766900 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:02.766851 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:17:06.011756 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:06.011720 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:17:06.779015 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:17:06.778976 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:17:07.436627 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:07.436596 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:17:09.970667 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:09.970636 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:17:14.249218 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:14.249188 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:17:36.121915 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:36.121868 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:17:36.778942 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:17:36.778904 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:17:40.833207 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:40.833169 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:17:45.523198 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:45.523153 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:17:46.511179 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:46.511140 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj"] Nov 01 07:17:46.511400 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:46.511373 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj" podUID="2eb5e419-7160-44b3-b69a-5f9fc85b6609" containerName="manager" containerID="cri-o://077fb3c71897f2cbc358b6a73eecda1861c1a14abe5589cea111099a8c4cd792" gracePeriod=10 Nov 01 07:17:46.759340 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:46.759315 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj" Nov 01 07:17:46.857300 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:46.857272 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eb5e419-7160-44b3-b69a-5f9fc85b6609-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2eb5e419-7160-44b3-b69a-5f9fc85b6609-extensions-socket-volume\") pod \"2eb5e419-7160-44b3-b69a-5f9fc85b6609\" (UID: \"2eb5e419-7160-44b3-b69a-5f9fc85b6609\") " Nov 01 07:17:46.857491 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:46.857330 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eb5e419-7160-44b3-b69a-5f9fc85b6609-kube-api-access-kjcmb\" (UniqueName: \"kubernetes.io/projected/2eb5e419-7160-44b3-b69a-5f9fc85b6609-kube-api-access-kjcmb\") pod \"2eb5e419-7160-44b3-b69a-5f9fc85b6609\" (UID: \"2eb5e419-7160-44b3-b69a-5f9fc85b6609\") " Nov 01 07:17:46.857715 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:46.857686 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb5e419-7160-44b3-b69a-5f9fc85b6609-extensions-socket-volume" pod "2eb5e419-7160-44b3-b69a-5f9fc85b6609" (UID: "2eb5e419-7160-44b3-b69a-5f9fc85b6609"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:17:46.859627 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:46.859600 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb5e419-7160-44b3-b69a-5f9fc85b6609-kube-api-access-kjcmb" pod "2eb5e419-7160-44b3-b69a-5f9fc85b6609" (UID: "2eb5e419-7160-44b3-b69a-5f9fc85b6609"). InnerVolumeSpecName "kube-api-access-kjcmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:17:46.958349 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:46.958317 4385 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2eb5e419-7160-44b3-b69a-5f9fc85b6609-extensions-socket-volume\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:17:46.958349 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:46.958346 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjcmb\" (UniqueName: \"kubernetes.io/projected/2eb5e419-7160-44b3-b69a-5f9fc85b6609-kube-api-access-kjcmb\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:17:47.616950 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:47.616909 4385 generic.go:356] "Generic (PLEG): container finished" podID="2eb5e419-7160-44b3-b69a-5f9fc85b6609" containerID="077fb3c71897f2cbc358b6a73eecda1861c1a14abe5589cea111099a8c4cd792" exitCode=0 Nov 01 07:17:47.617167 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:47.616963 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj" Nov 01 07:17:47.617167 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:47.617001 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj" event={"ID":"2eb5e419-7160-44b3-b69a-5f9fc85b6609","Type":"ContainerDied","Data":"077fb3c71897f2cbc358b6a73eecda1861c1a14abe5589cea111099a8c4cd792"} Nov 01 07:17:47.617167 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:47.617067 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj" event={"ID":"2eb5e419-7160-44b3-b69a-5f9fc85b6609","Type":"ContainerDied","Data":"ae7fdc88b3bf81d29b9f73c3cf93322c22a3ed07bdf3483e8e6e07858e2c396f"} Nov 01 07:17:47.617167 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:47.617097 4385 scope.go:122] "RemoveContainer" containerID="077fb3c71897f2cbc358b6a73eecda1861c1a14abe5589cea111099a8c4cd792" Nov 01 07:17:47.630968 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:47.630948 4385 scope.go:122] "RemoveContainer" containerID="077fb3c71897f2cbc358b6a73eecda1861c1a14abe5589cea111099a8c4cd792" Nov 01 07:17:47.631233 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:17:47.631205 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077fb3c71897f2cbc358b6a73eecda1861c1a14abe5589cea111099a8c4cd792\": container with ID starting with 077fb3c71897f2cbc358b6a73eecda1861c1a14abe5589cea111099a8c4cd792 not found: ID does not exist" containerID="077fb3c71897f2cbc358b6a73eecda1861c1a14abe5589cea111099a8c4cd792" Nov 01 07:17:47.631300 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:47.631234 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077fb3c71897f2cbc358b6a73eecda1861c1a14abe5589cea111099a8c4cd792"} err="failed to get container status \"077fb3c71897f2cbc358b6a73eecda1861c1a14abe5589cea111099a8c4cd792\": rpc error: code = NotFound desc = could not find container \"077fb3c71897f2cbc358b6a73eecda1861c1a14abe5589cea111099a8c4cd792\": container with ID starting with 077fb3c71897f2cbc358b6a73eecda1861c1a14abe5589cea111099a8c4cd792 not found: ID does not exist" Nov 01 07:17:47.793797 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:47.793765 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj"] Nov 01 07:17:47.844163 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:47.844136 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jpshj"] Nov 01 07:17:48.382638 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:48.382602 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eb5e419-7160-44b3-b69a-5f9fc85b6609" path="/var/lib/kubelet/pods/2eb5e419-7160-44b3-b69a-5f9fc85b6609/volumes" Nov 01 07:17:50.233692 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:50.233646 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:17:52.907860 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:52.907808 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:17:54.916464 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:54.916430 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:17:56.759391 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:56.759358 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:17:56.957477 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:56.957442 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:17:59.615998 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:17:59.615953 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:18:04.320624 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:04.320540 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:18:06.778734 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:18:06.778694 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:18:08.590445 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:08.590392 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:18:08.769192 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:08.769162 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:18:09.134397 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:09.134366 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:18:13.826898 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:13.826856 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:18:18.525911 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:18.525869 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:18:23.219471 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:23.219436 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:18:27.918622 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:27.918580 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:18:32.633878 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:32.633824 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:18:36.779160 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:18:36.779122 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:18:37.329312 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:37.329270 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:18:42.023483 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:42.023440 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:18:46.829513 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:46.829469 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:18:50.239181 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.239150 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b9zd5"] Nov 01 07:18:50.239608 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.239498 4385 cpu_manager.go:414] "containerMap: removing container" podUID="9a3802f6-96b7-4b16-9696-da0eeee659ef" containerName="cleanup" Nov 01 07:18:50.239608 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.239508 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a3802f6-96b7-4b16-9696-da0eeee659ef" containerName="cleanup" Nov 01 07:18:50.239608 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.239520 4385 cpu_manager.go:414] "containerMap: removing container" podUID="b74b50a6-08f2-4bc8-966a-1f23722f2807" containerName="istio-proxy" Nov 01 07:18:50.239608 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.239526 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b74b50a6-08f2-4bc8-966a-1f23722f2807" containerName="istio-proxy" Nov 01 07:18:50.239608 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.239538 4385 cpu_manager.go:414] "containerMap: removing container" podUID="9a3802f6-96b7-4b16-9696-da0eeee659ef" containerName="cleanup" Nov 01 07:18:50.239608 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.239543 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a3802f6-96b7-4b16-9696-da0eeee659ef" containerName="cleanup" Nov 01 07:18:50.239608 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.239568 4385 cpu_manager.go:414] "containerMap: removing container" podUID="2eb5e419-7160-44b3-b69a-5f9fc85b6609" containerName="manager" Nov 01 07:18:50.239608 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.239572 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb5e419-7160-44b3-b69a-5f9fc85b6609" containerName="manager" Nov 01 07:18:50.239945 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.239624 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="b74b50a6-08f2-4bc8-966a-1f23722f2807" containerName="istio-proxy" Nov 01 07:18:50.239945 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.239633 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a3802f6-96b7-4b16-9696-da0eeee659ef" containerName="cleanup" Nov 01 07:18:50.239945 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.239643 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb5e419-7160-44b3-b69a-5f9fc85b6609" containerName="manager" Nov 01 07:18:50.239945 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.239650 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a3802f6-96b7-4b16-9696-da0eeee659ef" containerName="cleanup" Nov 01 07:18:50.242563 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.242547 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b9zd5" Nov 01 07:18:50.295984 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.295943 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b9zd5"] Nov 01 07:18:50.336524 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.336494 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0aba0d23-1fbf-4648-97e7-e4916ee1c2d8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-b9zd5\" (UID: \"0aba0d23-1fbf-4648-97e7-e4916ee1c2d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b9zd5" Nov 01 07:18:50.336718 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.336534 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljgdc\" (UniqueName: \"kubernetes.io/projected/0aba0d23-1fbf-4648-97e7-e4916ee1c2d8-kube-api-access-ljgdc\") pod \"kuadrant-operator-controller-manager-678d87766f-b9zd5\" (UID: \"0aba0d23-1fbf-4648-97e7-e4916ee1c2d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b9zd5" Nov 01 07:18:50.342627 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.342609 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rbmtd\"" Nov 01 07:18:50.438178 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.438138 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0aba0d23-1fbf-4648-97e7-e4916ee1c2d8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-b9zd5\" (UID: \"0aba0d23-1fbf-4648-97e7-e4916ee1c2d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b9zd5" Nov 01 07:18:50.438178 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.438182 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljgdc\" (UniqueName: \"kubernetes.io/projected/0aba0d23-1fbf-4648-97e7-e4916ee1c2d8-kube-api-access-ljgdc\") pod \"kuadrant-operator-controller-manager-678d87766f-b9zd5\" (UID: \"0aba0d23-1fbf-4648-97e7-e4916ee1c2d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b9zd5" Nov 01 07:18:50.438505 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.438486 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0aba0d23-1fbf-4648-97e7-e4916ee1c2d8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-b9zd5\" (UID: \"0aba0d23-1fbf-4648-97e7-e4916ee1c2d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b9zd5" Nov 01 07:18:50.496770 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.496693 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljgdc\" (UniqueName: \"kubernetes.io/projected/0aba0d23-1fbf-4648-97e7-e4916ee1c2d8-kube-api-access-ljgdc\") pod \"kuadrant-operator-controller-manager-678d87766f-b9zd5\" (UID: \"0aba0d23-1fbf-4648-97e7-e4916ee1c2d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b9zd5" Nov 01 07:18:50.605632 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.605600 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rbmtd\"" Nov 01 07:18:50.615665 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:50.615638 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b9zd5" Nov 01 07:18:51.095046 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:51.095021 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b9zd5"] Nov 01 07:18:51.097036 ip-10-0-140-68 kubenswrapper[4385]: W1101 07:18:51.097003 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aba0d23_1fbf_4648_97e7_e4916ee1c2d8.slice/crio-a63c80c1e1348efa01af3e2a9c2ccb8f0ab9a43314503f781cb7449abb8bfd10 WatchSource:0}: Error finding container a63c80c1e1348efa01af3e2a9c2ccb8f0ab9a43314503f781cb7449abb8bfd10: Status 404 returned error can't find the container with id a63c80c1e1348efa01af3e2a9c2ccb8f0ab9a43314503f781cb7449abb8bfd10 Nov 01 07:18:51.562520 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:51.562423 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:18:51.960625 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:51.960583 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b9zd5" event={"ID":"0aba0d23-1fbf-4648-97e7-e4916ee1c2d8","Type":"ContainerStarted","Data":"c164ef62b3064461cbf694af811eb8a6bdf7d4d89a164928001f6638b1ebfcb4"} Nov 01 07:18:51.960824 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:51.960651 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b9zd5" event={"ID":"0aba0d23-1fbf-4648-97e7-e4916ee1c2d8","Type":"ContainerStarted","Data":"a63c80c1e1348efa01af3e2a9c2ccb8f0ab9a43314503f781cb7449abb8bfd10"} Nov 01 07:18:51.960824 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:51.960755 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b9zd5" Nov 01 07:18:52.177866 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:52.177798 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b9zd5" podStartSLOduration=2.177779744 podStartE2EDuration="2.177779744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:18:52.126336552 +0000 UTC m=+4141.890136408" watchObservedRunningTime="2026-11-01 07:18:52.177779744 +0000 UTC m=+4141.941579601" Nov 01 07:18:56.323822 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:56.323787 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:18:57.183744 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:57.183713 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:18:57.377374 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:18:57.377336 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:19:01.031782 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:01.031748 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:19:02.968789 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:02.968755 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b9zd5" Nov 01 07:19:05.732131 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:05.732092 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:19:06.780747 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:19:06.780710 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:19:10.424282 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:10.424249 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:19:15.142352 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:15.142314 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:19:19.817888 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:19.817851 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:19:21.745896 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.745860 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m"] Nov 01 07:19:21.746409 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.746390 4385 cpu_manager.go:414] "containerMap: removing container" podUID="9a3802f6-96b7-4b16-9696-da0eeee659ef" containerName="cleanup" Nov 01 07:19:21.746458 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.746411 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a3802f6-96b7-4b16-9696-da0eeee659ef" containerName="cleanup" Nov 01 07:19:21.746494 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.746484 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a3802f6-96b7-4b16-9696-da0eeee659ef" containerName="cleanup" Nov 01 07:19:21.749539 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.749520 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.821908 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.821877 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/522db485-fa5b-4b08-89a6-57e29aab376c-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.822102 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.821930 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/522db485-fa5b-4b08-89a6-57e29aab376c-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.822102 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.822072 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/522db485-fa5b-4b08-89a6-57e29aab376c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.822183 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.822106 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm4w4\" (UniqueName: \"kubernetes.io/projected/522db485-fa5b-4b08-89a6-57e29aab376c-kube-api-access-fm4w4\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.822183 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.822129 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/522db485-fa5b-4b08-89a6-57e29aab376c-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.822183 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.822148 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/522db485-fa5b-4b08-89a6-57e29aab376c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.822288 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.822239 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/522db485-fa5b-4b08-89a6-57e29aab376c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.822323 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.822295 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/522db485-fa5b-4b08-89a6-57e29aab376c-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.822361 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.822318 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/522db485-fa5b-4b08-89a6-57e29aab376c-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.855310 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.855275 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m"] Nov 01 07:19:21.923908 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.923870 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/522db485-fa5b-4b08-89a6-57e29aab376c-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.924104 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.923927 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/522db485-fa5b-4b08-89a6-57e29aab376c-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.924104 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.923971 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/522db485-fa5b-4b08-89a6-57e29aab376c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.924104 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.923989 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm4w4\" (UniqueName: \"kubernetes.io/projected/522db485-fa5b-4b08-89a6-57e29aab376c-kube-api-access-fm4w4\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.924104 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.924012 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/522db485-fa5b-4b08-89a6-57e29aab376c-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.924104 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.924029 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/522db485-fa5b-4b08-89a6-57e29aab376c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.924104 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.924072 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/522db485-fa5b-4b08-89a6-57e29aab376c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.924450 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.924124 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/522db485-fa5b-4b08-89a6-57e29aab376c-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.924450 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.924284 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/522db485-fa5b-4b08-89a6-57e29aab376c-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.924450 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.924357 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/522db485-fa5b-4b08-89a6-57e29aab376c-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.924450 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.924411 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/522db485-fa5b-4b08-89a6-57e29aab376c-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.924683 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.924502 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/522db485-fa5b-4b08-89a6-57e29aab376c-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.924683 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.924570 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/522db485-fa5b-4b08-89a6-57e29aab376c-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.924909 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.924887 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/522db485-fa5b-4b08-89a6-57e29aab376c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.926632 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.926615 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/522db485-fa5b-4b08-89a6-57e29aab376c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.926632 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.926622 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/522db485-fa5b-4b08-89a6-57e29aab376c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.980814 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.980777 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/522db485-fa5b-4b08-89a6-57e29aab376c-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:21.980991 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:21.980924 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm4w4\" (UniqueName: \"kubernetes.io/projected/522db485-fa5b-4b08-89a6-57e29aab376c-kube-api-access-fm4w4\") pod \"maas-default-gateway-openshift-default-7d9897b54c-2np4m\" (UID: \"522db485-fa5b-4b08-89a6-57e29aab376c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:22.062404 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:22.062321 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:22.491317 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:22.491290 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m"] Nov 01 07:19:22.492641 ip-10-0-140-68 kubenswrapper[4385]: W1101 07:19:22.492612 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522db485_fa5b_4b08_89a6_57e29aab376c.slice/crio-bd364f28cbee9e9a217580c1f523a944a202155f4027eb0f82017b69c8b80199 WatchSource:0}: Error finding container bd364f28cbee9e9a217580c1f523a944a202155f4027eb0f82017b69c8b80199: Status 404 returned error can't find the container with id bd364f28cbee9e9a217580c1f523a944a202155f4027eb0f82017b69c8b80199 Nov 01 07:19:22.495030 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:22.495002 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:19:22.495114 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:22.495063 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:19:22.495114 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:22.495092 4385 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 01 07:19:23.144455 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:23.144415 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" event={"ID":"522db485-fa5b-4b08-89a6-57e29aab376c","Type":"ContainerStarted","Data":"edb9d1b672ed57b995d78ffae4d43ac624cb55c2fefdaad7930c1a219d067a48"} Nov 01 07:19:23.144882 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:23.144480 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" event={"ID":"522db485-fa5b-4b08-89a6-57e29aab376c","Type":"ContainerStarted","Data":"bd364f28cbee9e9a217580c1f523a944a202155f4027eb0f82017b69c8b80199"} Nov 01 07:19:23.359867 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:23.359794 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" podStartSLOduration=2.359777352 podStartE2EDuration="2.359777352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:19:23.309275715 +0000 UTC m=+4173.073075573" watchObservedRunningTime="2026-11-01 07:19:23.359777352 +0000 UTC m=+4173.123577252" Nov 01 07:19:24.063194 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:24.063153 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:24.064629 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:24.064603 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2np4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 01 07:19:24.064740 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:24.064663 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" podUID="522db485-fa5b-4b08-89a6-57e29aab376c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 01 07:19:24.515394 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:24.515365 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:19:25.063597 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:25.063558 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2np4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 01 07:19:25.063783 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:25.063612 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" podUID="522db485-fa5b-4b08-89a6-57e29aab376c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 01 07:19:26.062770 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:26.062736 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2np4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 01 07:19:26.063226 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:26.062791 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" podUID="522db485-fa5b-4b08-89a6-57e29aab376c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 01 07:19:27.062980 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:27.062944 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2np4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 01 07:19:27.063426 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:27.063009 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" podUID="522db485-fa5b-4b08-89a6-57e29aab376c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 01 07:19:28.063690 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:28.063655 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2np4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 01 07:19:28.064225 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:28.063708 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" podUID="522db485-fa5b-4b08-89a6-57e29aab376c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 01 07:19:29.063562 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:29.063524 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2np4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Nov 01 07:19:29.063755 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:29.063578 4385 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" podUID="522db485-fa5b-4b08-89a6-57e29aab376c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Nov 01 07:19:29.207950 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:29.207908 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:19:30.821181 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:30.821143 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:30.822371 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:30.822353 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:30.823228 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:30.823208 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" Nov 01 07:19:33.934075 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:33.933963 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:19:36.778295 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:19:36.778256 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:19:38.614802 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:38.614752 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:19:43.410014 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:43.409977 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:19:48.130933 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:48.130885 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:19:50.748474 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:50.748440 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:19:50.943734 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:50.943694 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:19:52.825565 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:52.825529 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:19:57.516826 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:57.516790 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:19:57.658653 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:19:57.658619 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:20:01.530034 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:20:01.529985 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8ckbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:20:01.530688 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:20:01.530080 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" podUID="bf4f43cd-9da8-4bd6-9b49-9c7329b5480a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:20:02.429964 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:20:02.429936 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:20:06.788291 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:20:06.788246 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:20:07.176612 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:20:07.176572 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:20:12.016884 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:20:12.016847 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:20:16.702756 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:20:16.702710 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:20:21.346363 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:20:21.346330 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:20:26.015374 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:20:26.015335 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:20:30.852106 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:20:30.851998 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:20:31.504213 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:20:31.504175 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8ckbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:20:31.504562 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:20:31.504529 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" podUID="bf4f43cd-9da8-4bd6-9b49-9c7329b5480a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:20:35.613219 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:20:35.613184 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:20:36.780093 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:20:36.780052 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:20:40.332646 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:20:40.332615 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:20:45.029187 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:20:45.029156 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:20:49.727247 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:20:49.727207 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:20:51.973392 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:20:51.973357 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:20:52.156188 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:20:52.156155 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:20:54.420754 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:20:54.420716 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:20:57.852704 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:20:57.852666 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:20:59.128354 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:20:59.128321 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:21:01.540134 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:21:01.540009 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8ckbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:21:01.540925 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:21:01.540874 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" podUID="bf4f43cd-9da8-4bd6-9b49-9c7329b5480a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:21:01.872982 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:21:01.872947 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2np4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:21:01.873240 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:21:01.873212 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" podUID="522db485-fa5b-4b08-89a6-57e29aab376c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:21:03.837673 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:21:03.837639 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:21:06.778777 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:21:06.778733 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:21:08.532746 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:21:08.532710 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:21:13.318322 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:21:13.318274 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:21:18.040713 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:21:18.040676 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:21:22.749555 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:21:22.749514 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:21:27.534309 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:21:27.534264 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:21:32.231362 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:21:32.231333 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:21:36.778685 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:21:36.778645 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:21:37.009589 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:21:37.009547 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:21:41.729410 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:21:41.729371 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:21:46.610235 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:21:46.610195 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:21:51.315312 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:21:51.315264 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:21:53.071129 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:21:53.071085 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:21:53.280560 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:21:53.280524 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:21:56.168964 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:21:56.168927 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:21:58.193823 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:21:58.193780 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:21:58.384119 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:21:58.384082 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:00.838231 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:00.838190 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:03.597647 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:03.597609 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:03.744807 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:03.744774 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:05.522983 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:05.522806 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:06.844488 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:22:06.844423 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:22:10.316288 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:10.316246 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:15.028034 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:15.028000 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:19.729499 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:19.729466 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:24.414649 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:24.414616 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:29.208827 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:29.208793 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:33.923118 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:33.923020 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:36.423614 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:36.423574 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:36.780151 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:22:36.780066 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:22:38.248861 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:38.245081 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:38.401265 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:38.401227 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:38.755938 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:38.755873 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:41.373550 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:41.373512 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9"] Nov 01 07:22:41.540408 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:41.540370 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:43.391142 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:43.391102 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:43.592930 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:43.592884 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:48.261921 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:48.261885 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:49.388887 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:49.388825 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:50.314118 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:50.314086 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:50.463463 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:50.463396 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:54.933914 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:54.933873 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:22:56.393726 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.393664 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" podUID="db4341b8-4e10-4138-ae76-c4b4f8204a88" containerName="main" containerID="cri-o://dd9f6fe037a3b4219987f1654032efbbeb2ca24c2a70c275f897ebff873037c3" gracePeriod=45 Nov 01 07:22:56.587202 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.587167 4385 generic.go:356] "Generic (PLEG): container finished" podID="db4341b8-4e10-4138-ae76-c4b4f8204a88" containerID="dd9f6fe037a3b4219987f1654032efbbeb2ca24c2a70c275f897ebff873037c3" exitCode=0 Nov 01 07:22:56.587408 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.587254 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" event={"ID":"db4341b8-4e10-4138-ae76-c4b4f8204a88","Type":"ContainerDied","Data":"dd9f6fe037a3b4219987f1654032efbbeb2ca24c2a70c275f897ebff873037c3"} Nov 01 07:22:56.649359 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.649303 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:22:56.718147 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.718114 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-tmp-dir\") pod \"db4341b8-4e10-4138-ae76-c4b4f8204a88\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " Nov 01 07:22:56.718337 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.718157 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-model-cache\") pod \"db4341b8-4e10-4138-ae76-c4b4f8204a88\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " Nov 01 07:22:56.718337 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.718189 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4341b8-4e10-4138-ae76-c4b4f8204a88-kube-api-access-qfmdn\" (UniqueName: \"kubernetes.io/projected/db4341b8-4e10-4138-ae76-c4b4f8204a88-kube-api-access-qfmdn\") pod \"db4341b8-4e10-4138-ae76-c4b4f8204a88\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " Nov 01 07:22:56.718337 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.718215 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-dshm\" (UniqueName: \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-dshm\") pod \"db4341b8-4e10-4138-ae76-c4b4f8204a88\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " Nov 01 07:22:56.718337 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.718325 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db4341b8-4e10-4138-ae76-c4b4f8204a88-tls-certs\" (UniqueName: \"kubernetes.io/secret/db4341b8-4e10-4138-ae76-c4b4f8204a88-tls-certs\") pod \"db4341b8-4e10-4138-ae76-c4b4f8204a88\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " Nov 01 07:22:56.718569 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.718413 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-home\" (UniqueName: \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-home\") pod \"db4341b8-4e10-4138-ae76-c4b4f8204a88\" (UID: \"db4341b8-4e10-4138-ae76-c4b4f8204a88\") " Nov 01 07:22:56.718569 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.718452 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-tmp-dir" pod "db4341b8-4e10-4138-ae76-c4b4f8204a88" (UID: "db4341b8-4e10-4138-ae76-c4b4f8204a88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:22:56.718569 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.718468 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-model-cache" pod "db4341b8-4e10-4138-ae76-c4b4f8204a88" (UID: "db4341b8-4e10-4138-ae76-c4b4f8204a88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:22:56.718739 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.718662 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-home" pod "db4341b8-4e10-4138-ae76-c4b4f8204a88" (UID: "db4341b8-4e10-4138-ae76-c4b4f8204a88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:22:56.718739 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.718730 4385 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-home\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:22:56.718892 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.718745 4385 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-tmp-dir\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:22:56.718892 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.718756 4385 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-model-cache\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:22:56.720635 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.720601 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-dshm" pod "db4341b8-4e10-4138-ae76-c4b4f8204a88" (UID: "db4341b8-4e10-4138-ae76-c4b4f8204a88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:22:56.720740 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.720655 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4341b8-4e10-4138-ae76-c4b4f8204a88-kube-api-access-qfmdn" pod "db4341b8-4e10-4138-ae76-c4b4f8204a88" (UID: "db4341b8-4e10-4138-ae76-c4b4f8204a88"). InnerVolumeSpecName "kube-api-access-qfmdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:22:56.720740 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.720676 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db4341b8-4e10-4138-ae76-c4b4f8204a88-tls-certs" pod "db4341b8-4e10-4138-ae76-c4b4f8204a88" (UID: "db4341b8-4e10-4138-ae76-c4b4f8204a88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:22:56.819961 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.819929 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfmdn\" (UniqueName: \"kubernetes.io/projected/db4341b8-4e10-4138-ae76-c4b4f8204a88-kube-api-access-qfmdn\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:22:56.819961 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.819960 4385 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db4341b8-4e10-4138-ae76-c4b4f8204a88-dshm\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:22:56.820171 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:56.819972 4385 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db4341b8-4e10-4138-ae76-c4b4f8204a88-tls-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:22:57.594249 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:57.594216 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" Nov 01 07:22:57.594754 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:57.594259 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9" event={"ID":"db4341b8-4e10-4138-ae76-c4b4f8204a88","Type":"ContainerDied","Data":"5eab586043399fa01137276a3b40f282decc722c57671c4ad4fe650e82b0229e"} Nov 01 07:22:57.594754 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:57.594309 4385 scope.go:122] "RemoveContainer" containerID="dd9f6fe037a3b4219987f1654032efbbeb2ca24c2a70c275f897ebff873037c3" Nov 01 07:22:57.757158 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:57.757127 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9"] Nov 01 07:22:57.806438 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:57.806402 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-main-976689/simulator-main-976689-kserve-5c6db9ddfb-nxhv9"] Nov 01 07:22:58.385252 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:58.385214 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db4341b8-4e10-4138-ae76-c4b4f8204a88" path="/var/lib/kubelet/pods/db4341b8-4e10-4138-ae76-c4b4f8204a88/volumes" Nov 01 07:22:58.562456 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:22:58.562421 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:00.400084 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:00.400052 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:02.793616 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:02.793559 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:05.797891 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:05.797859 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:05.950563 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:05.950525 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:06.779272 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:23:06.779235 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:23:17.988176 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:17.988132 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:18.174003 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:18.173971 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:18.817266 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:18.817234 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:20.402428 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:20.402393 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:21.033420 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:21.033382 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:24.324072 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:24.324042 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:24.523908 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:24.523867 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:25.296408 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:25.296373 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:25.878355 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:25.878314 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:26.066501 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:26.066465 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:29.028265 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:29.028222 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:29.187738 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:29.187704 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:29.347768 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:29.347738 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:32.211714 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:32.211677 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:32.412496 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:32.412462 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:35.283276 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:35.283232 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:35.470857 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:35.470801 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:35.609389 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:35.609357 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:36.778359 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:23:36.778322 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:23:40.117127 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:40.117095 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:44.926819 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:44.926786 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:49.632902 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:49.632860 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:54.712853 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:54.712796 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:58.878828 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:58.878788 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:23:59.775546 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:23:59.775514 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:24:04.618677 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:24:04.618583 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:24:06.778496 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:24:06.778460 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:24:09.327273 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:24:09.327238 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:24:14.028974 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:24:14.028939 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:24:17.435518 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:24:17.435485 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:24:17.580620 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:24:17.580588 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:24:18.735723 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:24:18.735678 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:24:22.840682 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:24:22.840641 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:24:24.313206 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:24:24.313160 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:24:24.592038 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:24:24.592003 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:24:26.692037 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:24:26.691998 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:24:36.778755 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:24:36.778702 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:24:59.292137 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:24:59.292105 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:25:06.834105 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:25:06.834066 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:25:36.779460 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:25:36.779375 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:25:57.108859 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.107206 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr"] Nov 01 07:25:57.108859 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.107864 4385 cpu_manager.go:414] "containerMap: removing container" podUID="db4341b8-4e10-4138-ae76-c4b4f8204a88" containerName="main" Nov 01 07:25:57.108859 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.107885 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4341b8-4e10-4138-ae76-c4b4f8204a88" containerName="main" Nov 01 07:25:57.108859 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.108004 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4341b8-4e10-4138-ae76-c4b4f8204a88" containerName="main" Nov 01 07:25:57.120868 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.114111 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:25:57.157589 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.157550 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr"] Nov 01 07:25:57.210871 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.210521 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 07:25:57.210871 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.210718 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 07:25:57.246046 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.244194 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-7a816b61-kserve-self-signed-certs\"" Nov 01 07:25:57.246408 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.246386 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hczhz\"" Nov 01 07:25:57.259202 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.258442 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgk4h\" (UniqueName: \"kubernetes.io/projected/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-kube-api-access-rgk4h\") pod \"e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:25:57.259202 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.258497 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-tmp-dir\") pod \"e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:25:57.259202 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.258528 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-dshm\") pod \"e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:25:57.259202 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.258557 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-model-cache\") pod \"e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:25:57.259202 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.258671 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-home\") pod \"e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:25:57.259202 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.258712 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-tls-certs\") pod \"e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:25:57.262775 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.262607 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:25:57.360692 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.360565 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-tmp-dir\") pod \"e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:25:57.360692 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.360669 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-dshm\") pod \"e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:25:57.361070 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.360712 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-model-cache\") pod \"e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:25:57.361070 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.360924 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-home\") pod \"e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:25:57.361070 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.360972 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-tls-certs\") pod \"e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:25:57.361070 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.361029 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgk4h\" (UniqueName: \"kubernetes.io/projected/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-kube-api-access-rgk4h\") pod \"e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:25:57.361329 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.361299 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-tmp-dir\") pod \"e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:25:57.361731 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.361706 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-model-cache\") pod \"e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:25:57.362117 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.361798 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-home\") pod \"e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:25:57.363687 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.363652 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-dshm\") pod \"e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:25:57.364007 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.363977 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-tls-certs\") pod \"e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:25:57.418908 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.418872 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgk4h\" (UniqueName: \"kubernetes.io/projected/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-kube-api-access-rgk4h\") pod \"e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:25:57.465644 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.465600 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:25:57.841770 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:57.841741 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr"] Nov 01 07:25:57.844364 ip-10-0-140-68 kubenswrapper[4385]: W1101 07:25:57.844335 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d7b0e77_8f4a_4b74_82ea_7a1baf332204.slice/crio-10216d379934c92afc34d6b856859b95727590b4cc025f78a6d90d87eaa2286b WatchSource:0}: Error finding container 10216d379934c92afc34d6b856859b95727590b4cc025f78a6d90d87eaa2286b: Status 404 returned error can't find the container with id 10216d379934c92afc34d6b856859b95727590b4cc025f78a6d90d87eaa2286b Nov 01 07:25:58.498191 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:58.498147 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:25:58.634517 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:58.634485 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" event={"ID":"2d7b0e77-8f4a-4b74-82ea-7a1baf332204","Type":"ContainerStarted","Data":"502e310dc6a01f0eb3c511eb572a64f59945f3a8955bcdfb422380cc12b8f4e7"} Nov 01 07:25:58.634721 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:58.634529 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" event={"ID":"2d7b0e77-8f4a-4b74-82ea-7a1baf332204","Type":"ContainerStarted","Data":"10216d379934c92afc34d6b856859b95727590b4cc025f78a6d90d87eaa2286b"} Nov 01 07:25:58.748665 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:58.748570 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" podStartSLOduration=1.748553381 podStartE2EDuration="1.748553381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:25:58.746493129 +0000 UTC m=+4568.510292986" watchObservedRunningTime="2026-11-01 07:25:58.748553381 +0000 UTC m=+4568.512353240" Nov 01 07:25:59.466312 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:59.466275 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:25:59.643028 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:25:59.642993 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:26:01.741862 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:26:01.740937 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:26:06.780874 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:26:06.780808 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:26:07.466522 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:26:07.466470 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:26:07.479701 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:26:07.479673 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:26:07.706408 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:26:07.706375 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:26:30.115089 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:26:30.115045 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:26:36.778858 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:26:36.778795 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:26:59.977473 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:26:59.977438 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:27:04.666612 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:04.666526 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:27:06.778290 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:27:06.778254 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:27:10.348491 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:10.348450 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr"] Nov 01 07:27:10.368479 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:10.368440 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:27:10.524729 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:10.524685 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:27:25.370346 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:25.370283 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" podUID="2d7b0e77-8f4a-4b74-82ea-7a1baf332204" containerName="main" containerID="cri-o://502e310dc6a01f0eb3c511eb572a64f59945f3a8955bcdfb422380cc12b8f4e7" gracePeriod=45 Nov 01 07:27:25.630397 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:25.630334 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:27:25.713424 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:25.713384 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-home\" (UniqueName: \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-home\") pod \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " Nov 01 07:27:25.713629 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:25.713439 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-dshm\") pod \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " Nov 01 07:27:25.713629 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:25.713495 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-model-cache\") pod \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " Nov 01 07:27:25.713629 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:25.713525 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-kube-api-access-rgk4h\" (UniqueName: \"kubernetes.io/projected/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-kube-api-access-rgk4h\") pod \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " Nov 01 07:27:25.713629 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:25.713586 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-tls-certs\") pod \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " Nov 01 07:27:25.713629 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:25.713614 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-tmp-dir\") pod \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\" (UID: \"2d7b0e77-8f4a-4b74-82ea-7a1baf332204\") " Nov 01 07:27:25.713902 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:25.713710 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-home" pod "2d7b0e77-8f4a-4b74-82ea-7a1baf332204" (UID: "2d7b0e77-8f4a-4b74-82ea-7a1baf332204"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:27:25.713902 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:25.713733 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-model-cache" pod "2d7b0e77-8f4a-4b74-82ea-7a1baf332204" (UID: "2d7b0e77-8f4a-4b74-82ea-7a1baf332204"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:27:25.714022 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:25.713973 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-tmp-dir" pod "2d7b0e77-8f4a-4b74-82ea-7a1baf332204" (UID: "2d7b0e77-8f4a-4b74-82ea-7a1baf332204"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:27:25.714022 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:25.713982 4385 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-home\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:27:25.714137 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:25.714028 4385 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-model-cache\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:27:25.716072 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:25.716047 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-tls-certs" pod "2d7b0e77-8f4a-4b74-82ea-7a1baf332204" (UID: "2d7b0e77-8f4a-4b74-82ea-7a1baf332204"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:27:25.716072 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:25.716058 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-dshm" pod "2d7b0e77-8f4a-4b74-82ea-7a1baf332204" (UID: "2d7b0e77-8f4a-4b74-82ea-7a1baf332204"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:27:25.716238 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:25.716074 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-kube-api-access-rgk4h" pod "2d7b0e77-8f4a-4b74-82ea-7a1baf332204" (UID: "2d7b0e77-8f4a-4b74-82ea-7a1baf332204"). InnerVolumeSpecName "kube-api-access-rgk4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:27:25.815071 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:25.815038 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgk4h\" (UniqueName: \"kubernetes.io/projected/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-kube-api-access-rgk4h\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:27:25.815071 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:25.815066 4385 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-tls-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:27:25.815071 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:25.815080 4385 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-tmp-dir\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:27:25.815331 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:25.815088 4385 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d7b0e77-8f4a-4b74-82ea-7a1baf332204-dshm\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:27:26.165868 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:26.165810 4385 generic.go:356] "Generic (PLEG): container finished" podID="2d7b0e77-8f4a-4b74-82ea-7a1baf332204" containerID="502e310dc6a01f0eb3c511eb572a64f59945f3a8955bcdfb422380cc12b8f4e7" exitCode=0 Nov 01 07:27:26.165868 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:26.165871 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" Nov 01 07:27:26.166170 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:26.165871 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" event={"ID":"2d7b0e77-8f4a-4b74-82ea-7a1baf332204","Type":"ContainerDied","Data":"502e310dc6a01f0eb3c511eb572a64f59945f3a8955bcdfb422380cc12b8f4e7"} Nov 01 07:27:26.166170 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:26.165923 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr" event={"ID":"2d7b0e77-8f4a-4b74-82ea-7a1baf332204","Type":"ContainerDied","Data":"10216d379934c92afc34d6b856859b95727590b4cc025f78a6d90d87eaa2286b"} Nov 01 07:27:26.166170 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:26.165947 4385 scope.go:122] "RemoveContainer" containerID="502e310dc6a01f0eb3c511eb572a64f59945f3a8955bcdfb422380cc12b8f4e7" Nov 01 07:27:26.175966 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:26.175951 4385 scope.go:122] "RemoveContainer" containerID="502e310dc6a01f0eb3c511eb572a64f59945f3a8955bcdfb422380cc12b8f4e7" Nov 01 07:27:26.176227 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:27:26.176203 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"502e310dc6a01f0eb3c511eb572a64f59945f3a8955bcdfb422380cc12b8f4e7\": container with ID starting with 502e310dc6a01f0eb3c511eb572a64f59945f3a8955bcdfb422380cc12b8f4e7 not found: ID does not exist" containerID="502e310dc6a01f0eb3c511eb572a64f59945f3a8955bcdfb422380cc12b8f4e7" Nov 01 07:27:26.176301 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:26.176236 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"502e310dc6a01f0eb3c511eb572a64f59945f3a8955bcdfb422380cc12b8f4e7"} err="failed to get container status \"502e310dc6a01f0eb3c511eb572a64f59945f3a8955bcdfb422380cc12b8f4e7\": rpc error: code = NotFound desc = could not find container \"502e310dc6a01f0eb3c511eb572a64f59945f3a8955bcdfb422380cc12b8f4e7\": container with ID starting with 502e310dc6a01f0eb3c511eb572a64f59945f3a8955bcdfb422380cc12b8f4e7 not found: ID does not exist" Nov 01 07:27:26.329087 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:26.329054 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr"] Nov 01 07:27:26.383350 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:26.383321 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-7a816b61-kserve-5594ff5697-ksmxr"] Nov 01 07:27:28.383199 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:28.383167 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d7b0e77-8f4a-4b74-82ea-7a1baf332204" path="/var/lib/kubelet/pods/2d7b0e77-8f4a-4b74-82ea-7a1baf332204/volumes" Nov 01 07:27:31.533861 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:31.532960 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8ckbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:27:31.533861 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:31.533031 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" podUID="bf4f43cd-9da8-4bd6-9b49-9c7329b5480a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:27:35.693356 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.693316 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4"] Nov 01 07:27:35.693918 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.693882 4385 cpu_manager.go:414] "containerMap: removing container" podUID="2d7b0e77-8f4a-4b74-82ea-7a1baf332204" containerName="main" Nov 01 07:27:35.693918 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.693900 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7b0e77-8f4a-4b74-82ea-7a1baf332204" containerName="main" Nov 01 07:27:35.694063 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.694026 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d7b0e77-8f4a-4b74-82ea-7a1baf332204" containerName="main" Nov 01 07:27:35.697076 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.697056 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:35.799972 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.799935 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hczhz\"" Nov 01 07:27:35.799972 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.799935 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 07:27:35.799972 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.799949 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 07:27:35.800291 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.800022 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-fff8f9af-kserve-self-signed-certs\"" Nov 01 07:27:35.802325 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.802292 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:27:35.809813 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.803587 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4"] Nov 01 07:27:35.822166 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.822140 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4z8f\" (UniqueName: \"kubernetes.io/projected/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-kube-api-access-w4z8f\") pod \"e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:35.822239 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.822181 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-home\") pod \"e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:35.822290 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.822271 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-model-cache\") pod \"e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:35.822327 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.822300 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-tmp-dir\") pod \"e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:35.822366 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.822342 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-tls-certs\") pod \"e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:35.822444 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.822421 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-dshm\") pod \"e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:35.923381 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.923348 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4z8f\" (UniqueName: \"kubernetes.io/projected/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-kube-api-access-w4z8f\") pod \"e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:35.923381 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.923395 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-home\") pod \"e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:35.923636 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.923444 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-model-cache\") pod \"e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:35.923636 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.923462 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-tmp-dir\") pod \"e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:35.923636 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.923510 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-tls-certs\") pod \"e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:35.923636 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.923549 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-dshm\") pod \"e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:35.923932 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.923909 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-model-cache\") pod \"e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:35.924018 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.923935 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-tmp-dir\") pod \"e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:35.924018 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.923988 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-home\") pod \"e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:35.926010 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.925989 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-dshm\") pod \"e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:35.926311 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.926295 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-tls-certs\") pod \"e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:35.979868 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:35.979780 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4z8f\" (UniqueName: \"kubernetes.io/projected/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-kube-api-access-w4z8f\") pod \"e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:36.008757 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:36.008730 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:36.409260 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:36.409228 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4"] Nov 01 07:27:36.410166 ip-10-0-140-68 kubenswrapper[4385]: W1101 07:27:36.410142 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3c48215_e6cf_4dd4_978b_8fbdc2eaa71b.slice/crio-3b36899b8a3be74271cc22f8b317f06b4e469bdb75e3424500b31df83c9c1db4 WatchSource:0}: Error finding container 3b36899b8a3be74271cc22f8b317f06b4e469bdb75e3424500b31df83c9c1db4: Status 404 returned error can't find the container with id 3b36899b8a3be74271cc22f8b317f06b4e469bdb75e3424500b31df83c9c1db4 Nov 01 07:27:36.780192 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:27:36.780106 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:27:37.272936 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:37.272899 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" event={"ID":"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b","Type":"ContainerStarted","Data":"dcbbe9e36a3d7da83f22bb29793665b0ebd62620983c1e4472d78ec1fed2cce5"} Nov 01 07:27:37.272936 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:37.272939 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" event={"ID":"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b","Type":"ContainerStarted","Data":"3b36899b8a3be74271cc22f8b317f06b4e469bdb75e3424500b31df83c9c1db4"} Nov 01 07:27:37.386912 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:37.386850 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" podStartSLOduration=2.386820009 podStartE2EDuration="2.386820009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:27:37.385052637 +0000 UTC m=+4667.148852494" watchObservedRunningTime="2026-11-01 07:27:37.386820009 +0000 UTC m=+4667.150619866" Nov 01 07:27:38.009033 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:38.008996 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:46.009094 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:46.009059 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:46.021553 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:46.021522 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:46.371993 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:46.371960 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:27:49.756601 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:49.756570 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:27:51.464977 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:51.464936 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:27:51.655282 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:27:51.655242 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:28:00.340301 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:28:00.340258 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:28:03.659206 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:28:03.659166 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:28:06.779015 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:28:06.778978 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:28:19.180599 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:28:19.180558 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:28:19.372124 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:28:19.372081 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:28:36.778616 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:28:36.778535 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:28:51.259315 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:28:51.259274 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4"] Nov 01 07:28:51.291715 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:28:51.291680 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:29:00.677896 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:00.677858 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:29:03.584314 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:03.584280 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:29:06.279008 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.278969 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" podUID="a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b" containerName="main" containerID="cri-o://dcbbe9e36a3d7da83f22bb29793665b0ebd62620983c1e4472d78ec1fed2cce5" gracePeriod=45 Nov 01 07:29:06.361226 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.361186 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" podUID="a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.174:8000/ready\": dial tcp 10.134.0.174:8000: connect: connection refused" Nov 01 07:29:06.531371 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.531313 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:29:06.616716 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.616683 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-home\" (UniqueName: \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-home\") pod \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " Nov 01 07:29:06.616931 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.616727 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-tmp-dir\") pod \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " Nov 01 07:29:06.616931 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.616763 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-dshm\") pod \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " Nov 01 07:29:06.616931 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.616804 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-tls-certs\" (UniqueName: \"kubernetes.io/secret/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-tls-certs\") pod \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " Nov 01 07:29:06.616931 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.616879 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-kube-api-access-w4z8f\" (UniqueName: \"kubernetes.io/projected/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-kube-api-access-w4z8f\") pod \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " Nov 01 07:29:06.616931 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.616907 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-model-cache\") pod \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\" (UID: \"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b\") " Nov 01 07:29:06.617233 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.617035 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-home" pod "a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b" (UID: "a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:29:06.617233 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.617047 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-tmp-dir" pod "a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b" (UID: "a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:29:06.617233 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.617180 4385 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-home\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:29:06.617233 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.617198 4385 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-tmp-dir\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:29:06.617681 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.617254 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-model-cache" pod "a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b" (UID: "a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:29:06.619227 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.619200 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-dshm" pod "a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b" (UID: "a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:29:06.619227 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.619212 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-tls-certs" pod "a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b" (UID: "a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:29:06.619391 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.619331 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-kube-api-access-w4z8f" pod "a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b" (UID: "a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b"). InnerVolumeSpecName "kube-api-access-w4z8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:29:06.718577 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.718539 4385 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-dshm\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:29:06.718577 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.718569 4385 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-tls-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:29:06.718577 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.718583 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4z8f\" (UniqueName: \"kubernetes.io/projected/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-kube-api-access-w4z8f\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:29:06.718824 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.718597 4385 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b-model-cache\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:29:06.778464 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:29:06.778429 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:29:06.899948 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.899915 4385 generic.go:356] "Generic (PLEG): container finished" podID="a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b" containerID="dcbbe9e36a3d7da83f22bb29793665b0ebd62620983c1e4472d78ec1fed2cce5" exitCode=0 Nov 01 07:29:06.900124 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.899964 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" Nov 01 07:29:06.900124 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.900004 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" event={"ID":"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b","Type":"ContainerDied","Data":"dcbbe9e36a3d7da83f22bb29793665b0ebd62620983c1e4472d78ec1fed2cce5"} Nov 01 07:29:06.900124 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.900051 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4" event={"ID":"a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b","Type":"ContainerDied","Data":"3b36899b8a3be74271cc22f8b317f06b4e469bdb75e3424500b31df83c9c1db4"} Nov 01 07:29:06.900124 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.900071 4385 scope.go:122] "RemoveContainer" containerID="dcbbe9e36a3d7da83f22bb29793665b0ebd62620983c1e4472d78ec1fed2cce5" Nov 01 07:29:06.909964 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.909948 4385 scope.go:122] "RemoveContainer" containerID="dcbbe9e36a3d7da83f22bb29793665b0ebd62620983c1e4472d78ec1fed2cce5" Nov 01 07:29:06.910242 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:29:06.910222 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcbbe9e36a3d7da83f22bb29793665b0ebd62620983c1e4472d78ec1fed2cce5\": container with ID starting with dcbbe9e36a3d7da83f22bb29793665b0ebd62620983c1e4472d78ec1fed2cce5 not found: ID does not exist" containerID="dcbbe9e36a3d7da83f22bb29793665b0ebd62620983c1e4472d78ec1fed2cce5" Nov 01 07:29:06.910311 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:06.910256 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcbbe9e36a3d7da83f22bb29793665b0ebd62620983c1e4472d78ec1fed2cce5"} err="failed to get container status \"dcbbe9e36a3d7da83f22bb29793665b0ebd62620983c1e4472d78ec1fed2cce5\": rpc error: code = NotFound desc = could not find container \"dcbbe9e36a3d7da83f22bb29793665b0ebd62620983c1e4472d78ec1fed2cce5\": container with ID starting with dcbbe9e36a3d7da83f22bb29793665b0ebd62620983c1e4472d78ec1fed2cce5 not found: ID does not exist" Nov 01 07:29:07.062679 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:07.062643 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4"] Nov 01 07:29:07.113726 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:07.113686 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-fff8f9af-kserve-775d49d8b9-dzrg4"] Nov 01 07:29:08.383031 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:29:08.382996 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b" path="/var/lib/kubelet/pods/a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b/volumes" Nov 01 07:29:36.781335 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:29:36.781294 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:30:00.973737 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:30:00.973689 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:30:06.778321 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:30:06.778280 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:30:30.188391 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:30:30.188353 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:30:30.629452 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:30:30.629405 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:30:33.305696 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:30:33.305658 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:30:36.852910 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:30:36.852871 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:31:01.371462 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:31:01.371425 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:31:06.778893 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:31:06.778827 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:31:36.779243 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:31:36.779131 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:32:01.606100 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:32:01.606051 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:32:06.779536 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:32:06.779497 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:32:27.193900 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:32:27.193856 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:32:27.388297 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:32:27.388263 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:32:27.708897 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:32:27.708858 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:32:31.884030 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:32:31.883978 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2np4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:32:31.884749 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:32:31.884705 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" podUID="522db485-fa5b-4b08-89a6-57e29aab376c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:32:36.780008 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:32:36.779974 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:32:41.689768 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:32:41.689731 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:33:03.106404 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:33:03.106317 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:33:06.778874 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:33:06.778819 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:33:36.779417 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:33:36.779382 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:34:02.318267 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:34:02.318233 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:34:06.778760 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:34:06.778727 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:34:36.787909 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:34:36.787803 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:34:51.995802 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:34:51.995762 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:35:02.653629 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:35:02.653585 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:35:06.779739 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:35:06.779699 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:35:36.779312 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:35:36.779277 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:35:47.391724 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:35:47.391688 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:36:03.077871 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:36:03.077762 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:36:06.778440 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:36:06.778373 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:36:36.778737 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:36:36.778700 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:37:03.284343 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:03.284301 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:03.813124 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:03.813093 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:04.355588 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:04.355553 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:04.576248 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:04.576194 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:06.777699 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:37:06.777654 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:37:08.896381 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:08.896350 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:09.070081 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:09.070047 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:09.223740 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:09.223652 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:12.130691 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:12.130654 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:12.296922 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:12.296890 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:15.065660 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:15.065618 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:15.226697 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:15.226661 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:15.422339 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:15.422305 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:19.924284 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:19.924241 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:24.630533 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:24.630492 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:29.321402 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:29.321361 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:33.755979 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:33.755892 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:34.236470 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:34.236430 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:34.385945 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:34.385913 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:34.537031 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:34.536951 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:36.780087 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:37:36.780053 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:37:37.081196 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.081158 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6"] Nov 01 07:37:37.081616 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.081597 4385 cpu_manager.go:414] "containerMap: removing container" podUID="a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b" containerName="main" Nov 01 07:37:37.081711 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.081619 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b" containerName="main" Nov 01 07:37:37.081769 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.081732 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c48215-e6cf-4dd4-978b-8fbdc2eaa71b" containerName="main" Nov 01 07:37:37.084816 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.084797 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:37.138217 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.137976 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-tmp-dir\") pod \"simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:37.138217 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.138158 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2p2s\" (UniqueName: \"kubernetes.io/projected/d96a6af1-0374-459b-9b24-cb1919e0a953-kube-api-access-f2p2s\") pod \"simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:37.138217 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.138190 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-dshm\") pod \"simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:37.138534 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.138255 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-model-cache\") pod \"simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:37.138534 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.138323 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-home\") pod \"simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:37.138534 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.138358 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d96a6af1-0374-459b-9b24-cb1919e0a953-tls-certs\") pod \"simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:37.185566 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.185532 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:37.187664 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.187642 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2624b0\"/\"openshift-service-ca.crt\"" Nov 01 07:37:37.187664 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.187657 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2624b0\"/\"kube-root-ca.crt\"" Nov 01 07:37:37.187820 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.187750 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2624b0\"/\"simulator-single-main-2624b0-kserve-self-signed-certs\"" Nov 01 07:37:37.188018 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.188002 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2624b0\"/\"default-dockercfg-6cqg6\"" Nov 01 07:37:37.190388 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.190372 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6"] Nov 01 07:37:37.239023 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.238994 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-tmp-dir\") pod \"simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:37.239227 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.239067 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2p2s\" (UniqueName: \"kubernetes.io/projected/d96a6af1-0374-459b-9b24-cb1919e0a953-kube-api-access-f2p2s\") pod \"simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:37.239227 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.239095 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-dshm\") pod \"simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:37.239227 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.239118 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-model-cache\") pod \"simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:37.239400 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.239327 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-home\") pod \"simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:37.239400 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.239362 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d96a6af1-0374-459b-9b24-cb1919e0a953-tls-certs\") pod \"simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:37.239519 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.239462 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-tmp-dir\") pod \"simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:37.239705 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.239683 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-home\") pod \"simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:37.239918 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.239901 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-model-cache\") pod \"simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:37.241559 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.241543 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-dshm\") pod \"simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:37.242024 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.241995 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d96a6af1-0374-459b-9b24-cb1919e0a953-tls-certs\") pod \"simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:37.295448 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.295425 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2p2s\" (UniqueName: \"kubernetes.io/projected/d96a6af1-0374-459b-9b24-cb1919e0a953-kube-api-access-f2p2s\") pod \"simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:37.395696 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.395619 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:37.772636 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.772612 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6"] Nov 01 07:37:37.773624 ip-10-0-140-68 kubenswrapper[4385]: W1101 07:37:37.773598 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd96a6af1_0374_459b_9b24_cb1919e0a953.slice/crio-b0766e0a5ae81118fde4868c2338458362a2b3f12eb208eeeff0c5d1d5e63da4 WatchSource:0}: Error finding container b0766e0a5ae81118fde4868c2338458362a2b3f12eb208eeeff0c5d1d5e63da4: Status 404 returned error can't find the container with id b0766e0a5ae81118fde4868c2338458362a2b3f12eb208eeeff0c5d1d5e63da4 Nov 01 07:37:37.998383 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.998348 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" event={"ID":"d96a6af1-0374-459b-9b24-cb1919e0a953","Type":"ContainerStarted","Data":"01ac2e73dcd6c884612e7479691ab20d7407c3b1344c93d2a6616f2f2f81e384"} Nov 01 07:37:37.998773 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:37.998390 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" event={"ID":"d96a6af1-0374-459b-9b24-cb1919e0a953","Type":"ContainerStarted","Data":"b0766e0a5ae81118fde4868c2338458362a2b3f12eb208eeeff0c5d1d5e63da4"} Nov 01 07:37:38.110550 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:38.110503 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" podStartSLOduration=1.110489273 podStartE2EDuration="1.110489273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:37:38.110283772 +0000 UTC m=+5267.874083630" watchObservedRunningTime="2026-11-01 07:37:38.110489273 +0000 UTC m=+5267.874289130" Nov 01 07:37:38.395963 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:38.395875 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:38.924347 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:38.924313 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:43.627955 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:43.627924 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:47.396371 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:47.396338 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:47.408486 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:47.408462 4385 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:48.066628 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:48.066600 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:37:48.330395 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:48.330363 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:53.035398 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:53.035355 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:37:57.707517 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:37:57.707482 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:38:02.419168 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:02.419128 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:38:03.655171 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:03.655134 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:38:06.780930 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:38:06.780882 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:38:07.110952 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:07.110920 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:38:11.824600 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:11.824565 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:38:16.512559 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:16.512527 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:38:21.219027 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:21.218993 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:38:25.930149 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:25.930111 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:38:30.620061 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:30.620030 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:38:33.748812 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:33.748779 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:38:34.196340 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:34.196296 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:38:35.323580 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:35.323547 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:38:36.781046 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:38:36.781002 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:38:38.696057 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:38.696020 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6"] Nov 01 07:38:38.790405 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:38.790372 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:38:39.452063 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:39.452030 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:38:39.642850 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:39.642761 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:38:39.781442 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:39.781339 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:38:40.297014 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:40.296975 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:38:42.738102 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:42.738067 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:38:45.794765 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:45.794725 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:38:45.998216 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:45.998183 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:38:53.717020 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:53.716956 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" podUID="d96a6af1-0374-459b-9b24-cb1919e0a953" containerName="main" containerID="cri-o://01ac2e73dcd6c884612e7479691ab20d7407c3b1344c93d2a6616f2f2f81e384" gracePeriod=45 Nov 01 07:38:53.969222 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:53.969161 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:38:54.033304 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.033279 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-dshm\") pod \"d96a6af1-0374-459b-9b24-cb1919e0a953\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " Nov 01 07:38:54.033430 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.033323 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-model-cache\") pod \"d96a6af1-0374-459b-9b24-cb1919e0a953\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " Nov 01 07:38:54.033430 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.033376 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-tmp-dir\") pod \"d96a6af1-0374-459b-9b24-cb1919e0a953\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " Nov 01 07:38:54.033430 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.033421 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d96a6af1-0374-459b-9b24-cb1919e0a953-tls-certs\" (UniqueName: \"kubernetes.io/secret/d96a6af1-0374-459b-9b24-cb1919e0a953-tls-certs\") pod \"d96a6af1-0374-459b-9b24-cb1919e0a953\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " Nov 01 07:38:54.033548 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.033451 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d96a6af1-0374-459b-9b24-cb1919e0a953-kube-api-access-f2p2s\" (UniqueName: \"kubernetes.io/projected/d96a6af1-0374-459b-9b24-cb1919e0a953-kube-api-access-f2p2s\") pod \"d96a6af1-0374-459b-9b24-cb1919e0a953\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " Nov 01 07:38:54.033548 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.033499 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-home\" (UniqueName: \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-home\") pod \"d96a6af1-0374-459b-9b24-cb1919e0a953\" (UID: \"d96a6af1-0374-459b-9b24-cb1919e0a953\") " Nov 01 07:38:54.033662 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.033628 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-tmp-dir" pod "d96a6af1-0374-459b-9b24-cb1919e0a953" (UID: "d96a6af1-0374-459b-9b24-cb1919e0a953"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:38:54.033662 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.033638 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-model-cache" pod "d96a6af1-0374-459b-9b24-cb1919e0a953" (UID: "d96a6af1-0374-459b-9b24-cb1919e0a953"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:38:54.033828 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.033806 4385 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-model-cache\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:38:54.033978 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.033849 4385 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-tmp-dir\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:38:54.033978 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.033859 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-home" pod "d96a6af1-0374-459b-9b24-cb1919e0a953" (UID: "d96a6af1-0374-459b-9b24-cb1919e0a953"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:38:54.035703 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.035682 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d96a6af1-0374-459b-9b24-cb1919e0a953-tls-certs" pod "d96a6af1-0374-459b-9b24-cb1919e0a953" (UID: "d96a6af1-0374-459b-9b24-cb1919e0a953"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:38:54.035703 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.035691 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-dshm" pod "d96a6af1-0374-459b-9b24-cb1919e0a953" (UID: "d96a6af1-0374-459b-9b24-cb1919e0a953"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:38:54.035876 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.035727 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96a6af1-0374-459b-9b24-cb1919e0a953-kube-api-access-f2p2s" pod "d96a6af1-0374-459b-9b24-cb1919e0a953" (UID: "d96a6af1-0374-459b-9b24-cb1919e0a953"). InnerVolumeSpecName "kube-api-access-f2p2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:38:54.135039 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.135011 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2p2s\" (UniqueName: \"kubernetes.io/projected/d96a6af1-0374-459b-9b24-cb1919e0a953-kube-api-access-f2p2s\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:38:54.135039 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.135037 4385 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-home\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:38:54.135039 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.135049 4385 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d96a6af1-0374-459b-9b24-cb1919e0a953-dshm\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:38:54.135270 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.135057 4385 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d96a6af1-0374-459b-9b24-cb1919e0a953-tls-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:38:54.414156 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.414118 4385 generic.go:356] "Generic (PLEG): container finished" podID="d96a6af1-0374-459b-9b24-cb1919e0a953" containerID="01ac2e73dcd6c884612e7479691ab20d7407c3b1344c93d2a6616f2f2f81e384" exitCode=0 Nov 01 07:38:54.414299 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.414166 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" Nov 01 07:38:54.414299 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.414197 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" event={"ID":"d96a6af1-0374-459b-9b24-cb1919e0a953","Type":"ContainerDied","Data":"01ac2e73dcd6c884612e7479691ab20d7407c3b1344c93d2a6616f2f2f81e384"} Nov 01 07:38:54.414299 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.414239 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6" event={"ID":"d96a6af1-0374-459b-9b24-cb1919e0a953","Type":"ContainerDied","Data":"b0766e0a5ae81118fde4868c2338458362a2b3f12eb208eeeff0c5d1d5e63da4"} Nov 01 07:38:54.414299 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.414260 4385 scope.go:122] "RemoveContainer" containerID="01ac2e73dcd6c884612e7479691ab20d7407c3b1344c93d2a6616f2f2f81e384" Nov 01 07:38:54.423353 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.423335 4385 scope.go:122] "RemoveContainer" containerID="01ac2e73dcd6c884612e7479691ab20d7407c3b1344c93d2a6616f2f2f81e384" Nov 01 07:38:54.423644 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:38:54.423624 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ac2e73dcd6c884612e7479691ab20d7407c3b1344c93d2a6616f2f2f81e384\": container with ID starting with 01ac2e73dcd6c884612e7479691ab20d7407c3b1344c93d2a6616f2f2f81e384 not found: ID does not exist" containerID="01ac2e73dcd6c884612e7479691ab20d7407c3b1344c93d2a6616f2f2f81e384" Nov 01 07:38:54.423692 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.423653 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ac2e73dcd6c884612e7479691ab20d7407c3b1344c93d2a6616f2f2f81e384"} err="failed to get container status \"01ac2e73dcd6c884612e7479691ab20d7407c3b1344c93d2a6616f2f2f81e384\": rpc error: code = NotFound desc = could not find container \"01ac2e73dcd6c884612e7479691ab20d7407c3b1344c93d2a6616f2f2f81e384\": container with ID starting with 01ac2e73dcd6c884612e7479691ab20d7407c3b1344c93d2a6616f2f2f81e384 not found: ID does not exist" Nov 01 07:38:54.577080 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.577050 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6"] Nov 01 07:38:54.581196 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:54.581175 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-2624b0/simulator-single-main-2624b0-kserve-79dc7f5bbb-xfcx6"] Nov 01 07:38:56.383592 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:38:56.383563 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d96a6af1-0374-459b-9b24-cb1919e0a953" path="/var/lib/kubelet/pods/d96a6af1-0374-459b-9b24-cb1919e0a953/volumes" Nov 01 07:39:03.986111 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:39:03.986035 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:39:06.778588 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:39:06.778548 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:39:13.856475 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:39:13.856437 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:39:14.158628 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:39:14.158544 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:39:14.906003 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:39:14.905943 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:39:16.112177 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:39:16.112134 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:39:16.315220 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:39:16.315184 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:39:16.468537 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:39:16.468456 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:39:20.925725 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:39:20.925691 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:39:21.811222 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:39:21.811187 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:39:22.008380 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:39:22.008339 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:39:25.627109 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:39:25.627077 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:39:30.235700 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:39:30.235662 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:39:34.917487 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:39:34.917448 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:39:36.778642 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:39:36.778607 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:39:39.511614 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:39:39.511575 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:39:43.623364 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:39:43.623328 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:39:45.083626 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:39:45.083593 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:39:45.351524 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:39:45.351435 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:39:45.958052 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:39:45.958002 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:40:04.328442 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:04.328410 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:40:06.778684 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:40:06.778648 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:40:24.507776 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:24.507744 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bb4d85776-g59j4"] Nov 01 07:40:24.511314 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:24.509240 4385 cpu_manager.go:414] "containerMap: removing container" podUID="d96a6af1-0374-459b-9b24-cb1919e0a953" containerName="main" Nov 01 07:40:24.511314 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:24.509269 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d96a6af1-0374-459b-9b24-cb1919e0a953" containerName="main" Nov 01 07:40:24.511314 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:24.509494 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="d96a6af1-0374-459b-9b24-cb1919e0a953" containerName="main" Nov 01 07:40:24.513609 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:24.513573 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" Nov 01 07:40:24.616793 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:24.616756 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 07:40:24.616793 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:24.616770 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 07:40:24.616793 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:24.616793 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-csmhq\"" Nov 01 07:40:24.619500 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:24.619479 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb4d85776-g59j4"] Nov 01 07:40:24.669938 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:24.669893 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-metrics-certs\") pod \"maas-api-bb4d85776-g59j4\" (UID: \"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" Nov 01 07:40:24.670162 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:24.670043 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6vnh\" (UniqueName: \"kubernetes.io/projected/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-kube-api-access-v6vnh\") pod \"maas-api-bb4d85776-g59j4\" (UID: \"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" Nov 01 07:40:24.670162 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:24.670067 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-openshift-service-ca\") pod \"maas-api-bb4d85776-g59j4\" (UID: \"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" Nov 01 07:40:24.670162 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:24.670127 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-maas-api-tls\") pod \"maas-api-bb4d85776-g59j4\" (UID: \"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" Nov 01 07:40:24.771185 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:24.771101 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6vnh\" (UniqueName: \"kubernetes.io/projected/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-kube-api-access-v6vnh\") pod \"maas-api-bb4d85776-g59j4\" (UID: \"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" Nov 01 07:40:24.771185 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:24.771133 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-openshift-service-ca\") pod \"maas-api-bb4d85776-g59j4\" (UID: \"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" Nov 01 07:40:24.771185 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:24.771154 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-maas-api-tls\") pod \"maas-api-bb4d85776-g59j4\" (UID: \"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" Nov 01 07:40:24.771503 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:24.771210 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-metrics-certs\") pod \"maas-api-bb4d85776-g59j4\" (UID: \"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" Nov 01 07:40:24.771907 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:24.771880 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-openshift-service-ca\") pod \"maas-api-bb4d85776-g59j4\" (UID: \"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" Nov 01 07:40:24.773819 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:24.773791 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-maas-api-tls\") pod \"maas-api-bb4d85776-g59j4\" (UID: \"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" Nov 01 07:40:24.773929 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:24.773870 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-metrics-certs\") pod \"maas-api-bb4d85776-g59j4\" (UID: \"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" Nov 01 07:40:24.829878 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:24.829827 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6vnh\" (UniqueName: \"kubernetes.io/projected/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-kube-api-access-v6vnh\") pod \"maas-api-bb4d85776-g59j4\" (UID: \"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" Nov 01 07:40:25.127392 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:25.127350 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" Nov 01 07:40:25.499669 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:25.499644 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb4d85776-g59j4"] Nov 01 07:40:25.500337 ip-10-0-140-68 kubenswrapper[4385]: W1101 07:40:25.500312 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc499e67e_08aa_4191_9e1d_d0a7b2fee8dc.slice/crio-592edb591835f714663b3b0caeb87fc5cae85804a21e8b18ff8e7769631f4369 WatchSource:0}: Error finding container 592edb591835f714663b3b0caeb87fc5cae85804a21e8b18ff8e7769631f4369: Status 404 returned error can't find the container with id 592edb591835f714663b3b0caeb87fc5cae85804a21e8b18ff8e7769631f4369 Nov 01 07:40:25.501633 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:25.501611 4385 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 07:40:25.945942 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:25.945907 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" event={"ID":"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc","Type":"ContainerStarted","Data":"592edb591835f714663b3b0caeb87fc5cae85804a21e8b18ff8e7769631f4369"} Nov 01 07:40:26.953444 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:26.953407 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" event={"ID":"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc","Type":"ContainerStarted","Data":"524d7c52e0701156eeb0b618659f703c455009041cf50f96848789946550cafd"} Nov 01 07:40:26.953898 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:26.953535 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" Nov 01 07:40:27.072568 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:27.072512 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" podStartSLOduration=2.323934809 podStartE2EDuration="3.072495825s" totalImagesPullingTime="748.561016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:40:27.069753757 +0000 UTC m=+5436.833553613" watchObservedRunningTime="2026-11-01 07:40:27.072495825 +0000 UTC m=+5436.836295682" Nov 01 07:40:32.964333 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:32.964264 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" Nov 01 07:40:33.069401 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.069369 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb4d85776-g59j4"] Nov 01 07:40:33.069589 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.069566 4385 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" podUID="c499e67e-08aa-4191-9e1d-d0a7b2fee8dc" containerName="maas-api" containerID="cri-o://524d7c52e0701156eeb0b618659f703c455009041cf50f96848789946550cafd" gracePeriod=30 Nov 01 07:40:33.079011 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.078984 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69"] Nov 01 07:40:33.083012 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.082989 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69" Nov 01 07:40:33.148290 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.148266 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee2a0453-6402-4944-890f-ee6574ebf589-metrics-certs\") pod \"maas-api-65578dc8d4-ngp69\" (UID: \"ee2a0453-6402-4944-890f-ee6574ebf589\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69" Nov 01 07:40:33.148425 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.148336 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee2a0453-6402-4944-890f-ee6574ebf589-openshift-service-ca\") pod \"maas-api-65578dc8d4-ngp69\" (UID: \"ee2a0453-6402-4944-890f-ee6574ebf589\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69" Nov 01 07:40:33.148482 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.148461 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvrnd\" (UniqueName: \"kubernetes.io/projected/ee2a0453-6402-4944-890f-ee6574ebf589-kube-api-access-kvrnd\") pod \"maas-api-65578dc8d4-ngp69\" (UID: \"ee2a0453-6402-4944-890f-ee6574ebf589\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69" Nov 01 07:40:33.148535 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.148523 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee2a0453-6402-4944-890f-ee6574ebf589-maas-api-tls\") pod \"maas-api-65578dc8d4-ngp69\" (UID: \"ee2a0453-6402-4944-890f-ee6574ebf589\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69" Nov 01 07:40:33.181932 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.181816 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69"] Nov 01 07:40:33.249462 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.249431 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee2a0453-6402-4944-890f-ee6574ebf589-openshift-service-ca\") pod \"maas-api-65578dc8d4-ngp69\" (UID: \"ee2a0453-6402-4944-890f-ee6574ebf589\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69" Nov 01 07:40:33.249610 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.249493 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvrnd\" (UniqueName: \"kubernetes.io/projected/ee2a0453-6402-4944-890f-ee6574ebf589-kube-api-access-kvrnd\") pod \"maas-api-65578dc8d4-ngp69\" (UID: \"ee2a0453-6402-4944-890f-ee6574ebf589\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69" Nov 01 07:40:33.249610 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.249543 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee2a0453-6402-4944-890f-ee6574ebf589-maas-api-tls\") pod \"maas-api-65578dc8d4-ngp69\" (UID: \"ee2a0453-6402-4944-890f-ee6574ebf589\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69" Nov 01 07:40:33.249738 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.249624 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee2a0453-6402-4944-890f-ee6574ebf589-metrics-certs\") pod \"maas-api-65578dc8d4-ngp69\" (UID: \"ee2a0453-6402-4944-890f-ee6574ebf589\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69" Nov 01 07:40:33.250291 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.250263 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee2a0453-6402-4944-890f-ee6574ebf589-openshift-service-ca\") pod \"maas-api-65578dc8d4-ngp69\" (UID: \"ee2a0453-6402-4944-890f-ee6574ebf589\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69" Nov 01 07:40:33.252863 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.252817 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee2a0453-6402-4944-890f-ee6574ebf589-metrics-certs\") pod \"maas-api-65578dc8d4-ngp69\" (UID: \"ee2a0453-6402-4944-890f-ee6574ebf589\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69" Nov 01 07:40:33.253059 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.253044 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee2a0453-6402-4944-890f-ee6574ebf589-maas-api-tls\") pod \"maas-api-65578dc8d4-ngp69\" (UID: \"ee2a0453-6402-4944-890f-ee6574ebf589\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69" Nov 01 07:40:33.305559 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.305534 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvrnd\" (UniqueName: \"kubernetes.io/projected/ee2a0453-6402-4944-890f-ee6574ebf589-kube-api-access-kvrnd\") pod \"maas-api-65578dc8d4-ngp69\" (UID: \"ee2a0453-6402-4944-890f-ee6574ebf589\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69" Nov 01 07:40:33.324820 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.324799 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" Nov 01 07:40:33.350599 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.350578 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-metrics-certs\") pod \"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc\" (UID: \"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc\") " Nov 01 07:40:33.350705 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.350636 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-maas-api-tls\") pod \"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc\" (UID: \"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc\") " Nov 01 07:40:33.350705 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.350698 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-openshift-service-ca\") pod \"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc\" (UID: \"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc\") " Nov 01 07:40:33.350828 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.350718 4385 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-kube-api-access-v6vnh\" (UniqueName: \"kubernetes.io/projected/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-kube-api-access-v6vnh\") pod \"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc\" (UID: \"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc\") " Nov 01 07:40:33.351068 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.351045 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-openshift-service-ca" pod "c499e67e-08aa-4191-9e1d-d0a7b2fee8dc" (UID: "c499e67e-08aa-4191-9e1d-d0a7b2fee8dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:40:33.352747 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.352720 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-metrics-certs" pod "c499e67e-08aa-4191-9e1d-d0a7b2fee8dc" (UID: "c499e67e-08aa-4191-9e1d-d0a7b2fee8dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:40:33.352822 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.352764 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-maas-api-tls" pod "c499e67e-08aa-4191-9e1d-d0a7b2fee8dc" (UID: "c499e67e-08aa-4191-9e1d-d0a7b2fee8dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:40:33.352822 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.352791 4385 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-kube-api-access-v6vnh" pod "c499e67e-08aa-4191-9e1d-d0a7b2fee8dc" (UID: "c499e67e-08aa-4191-9e1d-d0a7b2fee8dc"). InnerVolumeSpecName "kube-api-access-v6vnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:40:33.451664 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.451635 4385 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-metrics-certs\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:40:33.451664 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.451658 4385 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-maas-api-tls\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:40:33.451664 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.451670 4385 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-openshift-service-ca\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:40:33.451917 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.451679 4385 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6vnh\" (UniqueName: \"kubernetes.io/projected/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc-kube-api-access-v6vnh\") on node \"ip-10-0-140-68.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:40:33.453901 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.453886 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69" Nov 01 07:40:33.827747 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.827722 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69"] Nov 01 07:40:33.829299 ip-10-0-140-68 kubenswrapper[4385]: W1101 07:40:33.829270 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee2a0453_6402_4944_890f_ee6574ebf589.slice/crio-dbfe11706466a4f605c52d2d64ede2ae57fb2892c78db9317f68684dc60c0ee8 WatchSource:0}: Error finding container dbfe11706466a4f605c52d2d64ede2ae57fb2892c78db9317f68684dc60c0ee8: Status 404 returned error can't find the container with id dbfe11706466a4f605c52d2d64ede2ae57fb2892c78db9317f68684dc60c0ee8 Nov 01 07:40:33.990156 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.990130 4385 generic.go:356] "Generic (PLEG): container finished" podID="c499e67e-08aa-4191-9e1d-d0a7b2fee8dc" containerID="524d7c52e0701156eeb0b618659f703c455009041cf50f96848789946550cafd" exitCode=0 Nov 01 07:40:33.990559 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.990177 4385 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" Nov 01 07:40:33.990559 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.990224 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" event={"ID":"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc","Type":"ContainerDied","Data":"524d7c52e0701156eeb0b618659f703c455009041cf50f96848789946550cafd"} Nov 01 07:40:33.990559 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.990273 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb4d85776-g59j4" event={"ID":"c499e67e-08aa-4191-9e1d-d0a7b2fee8dc","Type":"ContainerDied","Data":"592edb591835f714663b3b0caeb87fc5cae85804a21e8b18ff8e7769631f4369"} Nov 01 07:40:33.990559 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.990299 4385 scope.go:122] "RemoveContainer" containerID="524d7c52e0701156eeb0b618659f703c455009041cf50f96848789946550cafd" Nov 01 07:40:33.991606 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.991586 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69" event={"ID":"ee2a0453-6402-4944-890f-ee6574ebf589","Type":"ContainerStarted","Data":"dbfe11706466a4f605c52d2d64ede2ae57fb2892c78db9317f68684dc60c0ee8"} Nov 01 07:40:33.999737 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:33.999721 4385 scope.go:122] "RemoveContainer" containerID="524d7c52e0701156eeb0b618659f703c455009041cf50f96848789946550cafd" Nov 01 07:40:34.000033 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:40:34.000014 4385 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"524d7c52e0701156eeb0b618659f703c455009041cf50f96848789946550cafd\": container with ID starting with 524d7c52e0701156eeb0b618659f703c455009041cf50f96848789946550cafd not found: ID does not exist" containerID="524d7c52e0701156eeb0b618659f703c455009041cf50f96848789946550cafd" Nov 01 07:40:34.000124 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:34.000043 4385 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524d7c52e0701156eeb0b618659f703c455009041cf50f96848789946550cafd"} err="failed to get container status \"524d7c52e0701156eeb0b618659f703c455009041cf50f96848789946550cafd\": rpc error: code = NotFound desc = could not find container \"524d7c52e0701156eeb0b618659f703c455009041cf50f96848789946550cafd\": container with ID starting with 524d7c52e0701156eeb0b618659f703c455009041cf50f96848789946550cafd not found: ID does not exist" Nov 01 07:40:34.152724 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:34.152693 4385 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb4d85776-g59j4"] Nov 01 07:40:34.156426 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:34.156405 4385 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb4d85776-g59j4"] Nov 01 07:40:34.383058 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:34.382988 4385 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c499e67e-08aa-4191-9e1d-d0a7b2fee8dc" path="/var/lib/kubelet/pods/c499e67e-08aa-4191-9e1d-d0a7b2fee8dc/volumes" Nov 01 07:40:34.999543 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:34.999510 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69" event={"ID":"ee2a0453-6402-4944-890f-ee6574ebf589","Type":"ContainerStarted","Data":"cd2eb2b308b1b9a8736c63a58c9dde442c22619853f3de021588c8590690f8f6"} Nov 01 07:40:34.999955 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:34.999647 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69" Nov 01 07:40:35.113252 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:35.113202 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69" podStartSLOduration=1.367857147 podStartE2EDuration="2.113186956s" totalImagesPullingTime="745.329809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:40:35.111743374 +0000 UTC m=+5444.875543232" watchObservedRunningTime="2026-11-01 07:40:35.113186956 +0000 UTC m=+5444.876986854" Nov 01 07:40:36.780286 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:40:36.780251 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:40:41.009762 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:40:41.009735 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-65578dc8d4-ngp69" Nov 01 07:41:04.707303 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:41:04.707259 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:41:06.780258 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:41:06.780217 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:41:36.784011 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:41:36.783972 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:42:02.581276 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:42:02.581185 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:42:04.995456 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:42:04.995418 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:42:06.786248 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:42:06.786211 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:42:36.783041 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:42:36.783003 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:42:49.804620 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:42:49.804583 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7ldgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:42:49.805407 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:42:49.805280 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7ldgk" podUID="f3893a1b-d9b9-432b-ac25-28df30013563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:43:05.335742 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:43:05.335707 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:43:06.785443 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:43:06.785371 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:43:31.531013 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:43:31.530920 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-8ckbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:43:31.531789 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:43:31.531758 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-8ckbd" podUID="bf4f43cd-9da8-4bd6-9b49-9c7329b5480a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:43:31.865015 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:43:31.864974 4385 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-2np4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:43:31.865209 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:43:31.865047 4385 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-2np4m" podUID="522db485-fa5b-4b08-89a6-57e29aab376c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:43:36.778707 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:43:36.778666 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:43:54.242565 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:43:54.242525 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58588b67c6-mr9nn_413fadb6-87d9-4270-b53e-a7bf64b32d60/authorino/0.log" Nov 01 07:44:05.669201 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:05.669165 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:44:06.512664 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:06.512626 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-k9s7l_34d1d6b5-e7a8-43b0-ba57-4ea70b9d1169/manager/0.log" Nov 01 07:44:06.777621 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:44:06.777540 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:44:07.323475 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:07.323440 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-446vb_57957a77-1fea-4f3d-a619-a655a33498be/manager/0.log" Nov 01 07:44:08.455731 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:08.455701 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-4nlhs_9b76cbc0-abbe-45e8-a061-e44e87c2f782/manager/0.log" Nov 01 07:44:09.797907 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:09.797878 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-65578dc8d4-ngp69_ee2a0453-6402-4944-890f-ee6574ebf589/maas-api/0.log" Nov 01 07:44:12.453612 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:12.453579 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-bgkgt_16f2382e-1541-4146-882a-44a663facc98/postgres/0.log" Nov 01 07:44:14.481405 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:14.481370 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58588b67c6-mr9nn_413fadb6-87d9-4270-b53e-a7bf64b32d60/authorino/0.log" Nov 01 07:44:14.890770 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:14.890735 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-6jbbb_1203942c-3f66-4dbc-b88a-7ab1ea9154be/manager/0.log" Nov 01 07:44:15.105444 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:15.105415 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-5hrk6_d6a699b6-b7e6-4e66-8a91-2db8f6209e11/manager/0.log" Nov 01 07:44:15.333498 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:15.333475 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pwpvc_045ea95f-c9de-4c1e-af56-ff3712203d6a/kuadrant-console-plugin/0.log" Nov 01 07:44:15.782868 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:15.782751 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-b9zd5_0aba0d23-1fbf-4648-97e7-e4916ee1c2d8/manager/0.log" Nov 01 07:44:16.259457 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:16.259413 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-q7mrd_79830bd7-36da-4336-86d4-8e63d1a7e440/manager/0.log" Nov 01 07:44:17.245911 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:17.245860 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd_59e2f40b-4458-49f6-a39a-93ad994e8aff/istio-proxy/0.log" Nov 01 07:44:18.310975 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:18.310945 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-2np4m_522db485-fa5b-4b08-89a6-57e29aab376c/istio-proxy/0.log" Nov 01 07:44:18.646670 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:18.646644 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-7ldgk_f3893a1b-d9b9-432b-ac25-28df30013563/istio-proxy/0.log" Nov 01 07:44:18.923651 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:18.923568 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-8ckbd_bf4f43cd-9da8-4bd6-9b49-9c7329b5480a/istio-proxy/1.log" Nov 01 07:44:21.204585 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:21.204555 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7_28f52bd7-a41a-4af8-9f66-911b3fe24b58/istio-proxy/0.log" Nov 01 07:44:22.668320 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:22.668291 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-8ckbd_bf4f43cd-9da8-4bd6-9b49-9c7329b5480a/istio-proxy/0.log" Nov 01 07:44:35.619644 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:35.619612 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:44:35.780531 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:35.780494 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:44:36.779199 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:44:36.779157 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:44:45.187017 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:45.186978 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-txvqp_5b0855c2-93a3-44a5-9311-63d891705e0e/global-pull-secret-syncer/0.log" Nov 01 07:44:46.911889 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:46.911862 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-k9s7l_34d1d6b5-e7a8-43b0-ba57-4ea70b9d1169/manager/0.log" Nov 01 07:44:47.841501 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:47.841468 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" Nov 01 07:44:48.028640 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:48.028603 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-446vb_57957a77-1fea-4f3d-a619-a655a33498be/manager/0.log" Nov 01 07:44:49.877201 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:49.877160 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:44:49.904170 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:49.904134 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-68.us-west-1.compute.internal_645c51de28f312aaa75d7c297b03488b/haproxy/0.log" Nov 01 07:44:50.363411 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:50.363381 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-4nlhs_9b76cbc0-abbe-45e8-a061-e44e87c2f782/manager/0.log" Nov 01 07:44:53.620933 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:53.620897 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-k9s7l_34d1d6b5-e7a8-43b0-ba57-4ea70b9d1169/manager/0.log" Nov 01 07:44:54.190165 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.190131 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58588b67c6-mr9nn_413fadb6-87d9-4270-b53e-a7bf64b32d60/authorino/0.log" Nov 01 07:44:54.297761 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.297726 4385 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth"] Nov 01 07:44:54.298337 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.298315 4385 cpu_manager.go:414] "containerMap: removing container" podUID="c499e67e-08aa-4191-9e1d-d0a7b2fee8dc" containerName="maas-api" Nov 01 07:44:54.298337 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.298337 4385 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c499e67e-08aa-4191-9e1d-d0a7b2fee8dc" containerName="maas-api" Nov 01 07:44:54.298540 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.298451 4385 memory_manager.go:365] "RemoveStaleState removing state" podUID="c499e67e-08aa-4191-9e1d-d0a7b2fee8dc" containerName="maas-api" Nov 01 07:44:54.301777 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.301752 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" Nov 01 07:44:54.353545 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.353515 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kd9k9\"/\"openshift-service-ca.crt\"" Nov 01 07:44:54.353787 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.353572 4385 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kd9k9\"/\"kube-root-ca.crt\"" Nov 01 07:44:54.403449 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.403425 4385 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kd9k9\"/\"default-dockercfg-hbgcf\"" Nov 01 07:44:54.406828 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.406792 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth"] Nov 01 07:44:54.414549 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.414522 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5l2v\" (UniqueName: \"kubernetes.io/projected/4fc17189-4476-4dd0-b39c-7814ac672ee2-kube-api-access-b5l2v\") pod \"perf-node-gather-daemonset-xhlth\" (UID: \"4fc17189-4476-4dd0-b39c-7814ac672ee2\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" Nov 01 07:44:54.414718 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.414603 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fc17189-4476-4dd0-b39c-7814ac672ee2-lib-modules\") pod \"perf-node-gather-daemonset-xhlth\" (UID: \"4fc17189-4476-4dd0-b39c-7814ac672ee2\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" Nov 01 07:44:54.414718 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.414627 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4fc17189-4476-4dd0-b39c-7814ac672ee2-podres\") pod \"perf-node-gather-daemonset-xhlth\" (UID: \"4fc17189-4476-4dd0-b39c-7814ac672ee2\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" Nov 01 07:44:54.414879 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.414863 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4fc17189-4476-4dd0-b39c-7814ac672ee2-proc\") pod \"perf-node-gather-daemonset-xhlth\" (UID: \"4fc17189-4476-4dd0-b39c-7814ac672ee2\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" Nov 01 07:44:54.414955 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.414934 4385 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fc17189-4476-4dd0-b39c-7814ac672ee2-sys\") pod \"perf-node-gather-daemonset-xhlth\" (UID: \"4fc17189-4476-4dd0-b39c-7814ac672ee2\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" Nov 01 07:44:54.515618 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.515513 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4fc17189-4476-4dd0-b39c-7814ac672ee2-proc\") pod \"perf-node-gather-daemonset-xhlth\" (UID: \"4fc17189-4476-4dd0-b39c-7814ac672ee2\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" Nov 01 07:44:54.515618 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.515577 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fc17189-4476-4dd0-b39c-7814ac672ee2-sys\") pod \"perf-node-gather-daemonset-xhlth\" (UID: \"4fc17189-4476-4dd0-b39c-7814ac672ee2\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" Nov 01 07:44:54.515898 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.515642 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5l2v\" (UniqueName: \"kubernetes.io/projected/4fc17189-4476-4dd0-b39c-7814ac672ee2-kube-api-access-b5l2v\") pod \"perf-node-gather-daemonset-xhlth\" (UID: \"4fc17189-4476-4dd0-b39c-7814ac672ee2\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" Nov 01 07:44:54.515898 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.515668 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4fc17189-4476-4dd0-b39c-7814ac672ee2-proc\") pod \"perf-node-gather-daemonset-xhlth\" (UID: \"4fc17189-4476-4dd0-b39c-7814ac672ee2\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" Nov 01 07:44:54.515898 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.515751 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fc17189-4476-4dd0-b39c-7814ac672ee2-lib-modules\") pod \"perf-node-gather-daemonset-xhlth\" (UID: \"4fc17189-4476-4dd0-b39c-7814ac672ee2\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" Nov 01 07:44:54.515898 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.515786 4385 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4fc17189-4476-4dd0-b39c-7814ac672ee2-podres\") pod \"perf-node-gather-daemonset-xhlth\" (UID: \"4fc17189-4476-4dd0-b39c-7814ac672ee2\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" Nov 01 07:44:54.516101 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.515909 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4fc17189-4476-4dd0-b39c-7814ac672ee2-podres\") pod \"perf-node-gather-daemonset-xhlth\" (UID: \"4fc17189-4476-4dd0-b39c-7814ac672ee2\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" Nov 01 07:44:54.516101 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.515930 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fc17189-4476-4dd0-b39c-7814ac672ee2-sys\") pod \"perf-node-gather-daemonset-xhlth\" (UID: \"4fc17189-4476-4dd0-b39c-7814ac672ee2\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" Nov 01 07:44:54.516101 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.515983 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fc17189-4476-4dd0-b39c-7814ac672ee2-lib-modules\") pod \"perf-node-gather-daemonset-xhlth\" (UID: \"4fc17189-4476-4dd0-b39c-7814ac672ee2\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" Nov 01 07:44:54.573849 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.573810 4385 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5l2v\" (UniqueName: \"kubernetes.io/projected/4fc17189-4476-4dd0-b39c-7814ac672ee2-kube-api-access-b5l2v\") pod \"perf-node-gather-daemonset-xhlth\" (UID: \"4fc17189-4476-4dd0-b39c-7814ac672ee2\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" Nov 01 07:44:54.618213 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.618179 4385 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" Nov 01 07:44:54.705560 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:54.705532 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-446vb_57957a77-1fea-4f3d-a619-a655a33498be/manager/0.log" Nov 01 07:44:55.006249 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:55.006222 4385 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth"] Nov 01 07:44:55.007870 ip-10-0-140-68 kubenswrapper[4385]: W1101 07:44:55.007818 4385 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4fc17189_4476_4dd0_b39c_7814ac672ee2.slice/crio-9e36a3e5a7cdd8436b46949560e2346ca7d09c02b5e4cf0bf1a07e6fa2a4c91f WatchSource:0}: Error finding container 9e36a3e5a7cdd8436b46949560e2346ca7d09c02b5e4cf0bf1a07e6fa2a4c91f: Status 404 returned error can't find the container with id 9e36a3e5a7cdd8436b46949560e2346ca7d09c02b5e4cf0bf1a07e6fa2a4c91f Nov 01 07:44:55.506294 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:55.506255 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-6jbbb_1203942c-3f66-4dbc-b88a-7ab1ea9154be/manager/0.log" Nov 01 07:44:55.662188 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:55.662139 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" event={"ID":"4fc17189-4476-4dd0-b39c-7814ac672ee2","Type":"ContainerStarted","Data":"afb9d24677fe668f688a894234c6730e1b6c4299f09530c12ad52d792bb9b840"} Nov 01 07:44:55.662409 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:55.662215 4385 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" event={"ID":"4fc17189-4476-4dd0-b39c-7814ac672ee2","Type":"ContainerStarted","Data":"9e36a3e5a7cdd8436b46949560e2346ca7d09c02b5e4cf0bf1a07e6fa2a4c91f"} Nov 01 07:44:55.662409 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:55.662256 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" Nov 01 07:44:55.776712 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:55.776565 4385 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" podStartSLOduration=1.776545324 podStartE2EDuration="1.776545324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 07:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:44:55.775679174 +0000 UTC m=+5705.539479033" watchObservedRunningTime="2026-11-01 07:44:55.776545324 +0000 UTC m=+5705.540345182" Nov 01 07:44:55.926765 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:55.926724 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-5hrk6_d6a699b6-b7e6-4e66-8a91-2db8f6209e11/manager/0.log" Nov 01 07:44:56.340749 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:56.340713 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pwpvc_045ea95f-c9de-4c1e-af56-ff3712203d6a/kuadrant-console-plugin/0.log" Nov 01 07:44:56.910172 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:56.910139 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-b9zd5_0aba0d23-1fbf-4648-97e7-e4916ee1c2d8/manager/0.log" Nov 01 07:44:57.065465 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:57.065434 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-4nlhs_9b76cbc0-abbe-45e8-a061-e44e87c2f782/manager/0.log" Nov 01 07:44:57.845666 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:44:57.845631 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-q7mrd_79830bd7-36da-4336-86d4-8e63d1a7e440/manager/0.log" Nov 01 07:45:00.404775 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:45:00.404736 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-k9s7l_34d1d6b5-e7a8-43b0-ba57-4ea70b9d1169/manager/0.log" Nov 01 07:45:01.591349 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:45:01.591249 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-446vb_57957a77-1fea-4f3d-a619-a655a33498be/manager/0.log" Nov 01 07:45:01.680877 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:45:01.680824 4385 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-xhlth" Nov 01 07:45:03.907626 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:45:03.907583 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-4nlhs_9b76cbc0-abbe-45e8-a061-e44e87c2f782/manager/0.log" Nov 01 07:45:06.008661 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:45:06.008617 4385 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8j7v_199a6bc2-3668-415c-8cc4-813f96eccfa4/konnectivity-agent/0.log" limit=1024 Nov 01 07:45:06.778716 ip-10-0-140-68 kubenswrapper[4385]: E1101 07:45:06.778684 4385 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9rlh" containerName="extractor" Nov 01 07:45:07.205789 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:45:07.205759 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tf6zg_23c957b7-d233-4fbe-a14e-4098f4229462/node-exporter/0.log" Nov 01 07:45:07.616120 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:45:07.616088 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tf6zg_23c957b7-d233-4fbe-a14e-4098f4229462/kube-rbac-proxy/0.log" Nov 01 07:45:07.798500 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:45:07.798461 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-858f55f4d9cr5hd_59e2f40b-4458-49f6-a39a-93ad994e8aff/istio-proxy/0.log" Nov 01 07:45:08.028580 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:45:08.028503 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tf6zg_23c957b7-d233-4fbe-a14e-4098f4229462/init-textfile/0.log" Nov 01 07:45:09.010901 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:45:09.010868 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-2np4m_522db485-fa5b-4b08-89a6-57e29aab376c/istio-proxy/0.log" Nov 01 07:45:09.224874 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:45:09.224846 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-7ldgk_f3893a1b-d9b9-432b-ac25-28df30013563/istio-proxy/0.log" Nov 01 07:45:09.437022 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:45:09.436994 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-8ckbd_bf4f43cd-9da8-4bd6-9b49-9c7329b5480a/istio-proxy/0.log" Nov 01 07:45:09.443246 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:45:09.443224 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-8ckbd_bf4f43cd-9da8-4bd6-9b49-9c7329b5480a/istio-proxy/1.log" Nov 01 07:45:11.046113 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:45:11.046083 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6dd6bfb767-bkcz7_28f52bd7-a41a-4af8-9f66-911b3fe24b58/istio-proxy/0.log" Nov 01 07:45:13.879798 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:45:13.879769 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-clhd7_23904dd3-3c84-4aba-aad2-6db71df3eda1/prometheus-operator/0.log" Nov 01 07:45:14.332825 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:45:14.332795 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-clhd7_23904dd3-3c84-4aba-aad2-6db71df3eda1/kube-rbac-proxy/0.log" Nov 01 07:45:14.747998 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:45:14.747910 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-6qjdb_703b9839-6695-4c5a-a243-77a2f350da29/prometheus-operator-admission-webhook/0.log" Nov 01 07:45:16.417925 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:45:16.417891 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56b4f84ff8-qfxnb_3ddc5224-736b-4cc5-8ed1-1cd16f35fc61/thanos-query/0.log" Nov 01 07:45:16.833273 ip-10-0-140-68 kubenswrapper[4385]: I1101 07:45:16.833236 4385 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56b4f84ff8-qfxnb_3ddc5224-736b-4cc5-8ed1-1cd16f35fc61/kube-rbac-proxy-web/0.log"